To install Stardog using apt-get run the following commands:

This will first add the Stardog gpg key to the systems and then fetch and install the latest Stardog deb package.

To install Stardog using yum run the following commands:

The packages require that OpenJDK 8 and all of its dependencies are installed on the system. The package managers will install them if they are not already there. Stardog is then configured to start on boot via systemd and thus it can be controlled by the systemctl tool as shown below:

To customize the environment in which stardog is run the file /etc/stardog.env.sh can be altered with key value pairs, for example:

The most important piece of configuration to do before you start Stardog is setting the STARDOG_HOME environment variable. This is the directory where all the Stardog databases and other files will be stored. If STARDOG_HOME is not defined, Stardog will use the Java user.dir property value. We recommend adding it to your ~./bash_profile, or if you are using a package manager, /etc/stardog.env.sh.

You should not set STARDOG_HOME to be the same as the directory where you put the Stardog binary. Our convention is to put Stardog in /opt/stardog/{$version} and set STARDOG_HOME to /var/stardog. If you are setting up Stardog for production or other serious usage, see Upgrading Stardog Server for additional guidance.

If you do not have a license key

You will be able to retrieve a trial license-key via the command line once you start Stardog.

If you have a license key

Add it to your STARDOG_HOME. Ensure that the stardog-license-key.bin file is readable by the Stardog process.

You can specify a different location for the license file by setting STARDOG_LICENSE_PATH.

Place the bin folder of the Stardog install on your PATH so the stardog and stardog-admin scripts can be used regardless of current working directory. We recommend adding to your ~./bash_profile (or if you are using a package manager, /etc/stardog.env.sh), though you can also set temporarily.

Before you get started, you should get Docker (if you don’t already have it) and select a port on your local machine for Studio to be available on in your browser(the steps below use port number 8888; make sure to substitute whatever number you’re going to use, if it’s a different one).

To get the latest version of in-browser Studio, perform the following steps:

At this point, you can stop and start the container whenever you need it, running docker stop stardog-studio and docker start stardog-studio, respectively (using whatever name you provided in step 3, above). The Docker Daemon and the Studio container must be running to access in-browser Studio. If Studio is not accessible please remember to start Docker, as it may not start automatically on startup.

To upgrade the in-browser version of Stardog Studio, simply open a terminal and run docker stop stardog-studio && docker rm stardog-studio (again, using whatever name you provided in step 3, above). Then, repeat steps 2 through 4, above.

In-browser Studio stores user data (e.g. saved connections, query history, open workspace tabs) in your browser’s localStorage, so you must use the same browser to access your persisted user data.

To install:

To enable, edit .bash\_profile:

First, you really should be using Homebrew…​ya heard?

If not, then:

Then, edit .bash\_profile:

And for our Linux friends:

Now put the Stardog autocomplete script—stardog-completion.sh—into your bash\_completion.d directory, typically one of /etc/bash_completion.d, /usr/local/etc/bash_completion.d or ~/bash_completion.d.

Alternately you can put it anywhere you want, but tell .bash_profile about it:

Stardog uses the value of the JVM argument java.io.tmpdir to write temporary files for many different operations. If you want to configure temp space to use a particular disk volume or partition, use the java.io.tmpdir JVM argument on Stardog startup.

Bad (or, at least, weird) things are guaranteed to happen if this part of the filesystem runs out of (or even low on) free disk space. Stardog will delete temporary files when they’re no longer needed. But Stardog admins should configure their monitoring systems to make sure that free disk space is always available, both on java.io.tmpdir and on the disk volume that hosts STARDOG_HOME.^[4]

The following twiddly knobs for Stardog Server are available in stardog.properties:^[5]

Monitoring information is available via the Java API, the HTTP API, the CLI or (if configured) the JMX interface. Performing a GET on /admin/status which will return a JSON object containing the information available the server and all the databases. This information is also available for Prometheus via /admin/status/prometheus, allowing Prometheus servers to scrape Stardog directly. The endpoint DB/status will return the monitoring information about the database status. The stardog-admin server status command will print a subset of this information on the console.

By default, JMX monitoring is not enabled. You can enable it by setting metrics.reporter=jmx in the stardog.properties file. Then, you can simply use a tool like VisualVM or JConsole to attach to the process running the JVM, or connect directly to the JMX server.

If you want to connect to the JMX server remotely you need to set metrics.jmx.remote.access=true in stardog.properties. Stardog will bind an RMI server for remote access on port 5833. If you want to change this port Stardog binds the remote server to, you can set the property metrics.jmx.port in stardog.properties.

Finally, if you wish to disable monitoring completely, set metrics.enabled to false in stardog.properties.

A database must be set to offline status before most configuration parameters may be changed. Hence, the normal routine is to set the database offline, change the parameters, and then set the database to online. All of these operations may be done programmatically from CLI tools, such that they can be scripted in advance to minimize downtime. In a future version, we will allow some properties to be set while the database remains online.

Databases are put online or offline synchronously: these operations block until other database activity is completed or terminated. See stardog-admin help db for details.

To set a database from online to offline:

To set the database online:

If Stardog Server is shutdown while a database is offline, the database will be offline when the server restarts.

Archetypes can be used as a predefined way of loading a schema and a set of constraints to the database just like any RDF data can be loaded to a database. These kinds of archetypes are called "inline" as their contents will appear in the database under predefined named graphs as explained next. These named graphs that are automatically created by archetypes can be queried and modified by the user as any other named graph.

Each archetype has a unique IRI identifying it and the schema contents of inline archetypes will be loaded into a named graph with that IRI. To see an example, follow the setup instructions to download the archetypes to ${STARDOG_HOME}/.archetypes and create a new database with the FOAF archetype:

If you query the database you will see a named graph automatically created:

Archetypes can also be defined in a "protected" mode where the schema and the constraints will be available for reasoning and validation services but they will not be stored in the database. In this mode, archetypes prevent unintended modifications to the schema and the constraints without losing their reasoning and validation functionality. An ontology like PROV is standardized by W3C and is not meant to change over time so the protected mode can be used with it.

The user-defined archetypes are inline by default but the archetype definition can be configured to make the schema and/or the constraints protected as explained in the Github repository.

The following example shows how using a protected archetype would look:

This example demonstrates that the database looks empty to regular SPARQL queries but reasoning queries see the PROV ontology. Similarly PROV constraints are visible for validation purposes but they cannot be removed by the icv drop command.

Before Stardog 7.2.0, the only way to define archetypes was by creating and registering a new Java class that contained the archetype definition. This method is deprecated as of Stardog 7.2.0 but it will continue to work until Stardog 8 at which point support for Java-based archetypes will be removed. Until that time, the Java-based PROV and SKOS archetypes that were bundled in the Stardog distribution as built-in archetypes will be available and can be used without setting up the archetype location as describe above.

To create a new database with the default options by simply providing a name and a set of initial datasets to load:

Datasets can be loaded later as well. To create (in this case, an empty) database from a template file:

At a minimum, the configuration file must have a value for database.name option.

If you only want to change only a few configuration options you can directly give the values for these options in the CLI args as follows:

“--” is used in this case when “-o” is the last option to delimit the value for “-o” from the files to be bulk loaded.

Please refer to the CLI help for more details of the db create command.

We explain database backups and server backups in the following sections.

Database backup saves the contents of a single database along with database metadata including user and role permissions associated with the database. The stardog-admin db backup command assumes a default location for its output, namely, $STARDOG_HOME/.backup; that default may be overriden by setting backup.dir. Backups are stored in directories by database name and then in date-versioned subdirectories for each backup volume.

If you need to specify a location outside of $STARDOG_HOME (e.g. a network mount) you can set backup.location or pass it to the --to argument.

To backup a Stardog database called foobar:

To perform a remote backup, for example, pass in a specific directory that may be mounted in the current OS namespace via some network protocol, thus:

Finally, database backups can also be performed directly to S3 or GCP. For S3 backups use a URL in the following format:

The endpoint hostname and endpoint port values are only used for on-premises S3 clones. To use Amazon S3 those values can be left blank and the URL will have three / before the bucket as in:

For GCP backups use a URL in the following format:

See [GCP documentation](https://cloud.google.com/docs/authentication/production) for creating Google credentials JSON file.

A default S3 or GCP backup location can also be specified in the stardog.properties file with the key backup.location.

Server backup will back up the entire Stardog server, all databases and associated metadata. Unlike database backups, which takes a full backup of the database every time it is run, server backup takes an incremental backup of the data. That way, each time the command is run only the updates to the databases since the last time backup need to be saved.

Server backups are accomplished with the following command:

You can optionally specify the backup location, otherwise the Stardog defaults will be used, similar to the db backup command:

This command only supports file-based backups, it cannot be used with S3, for instance.

To copy the backups off local disk we recommend using a tool such as rclone.

After setting up rclone you can use it to send the backups to another server:

Consult the rclone docs for the full list of supported storage systems.

There are different restore commands corresponding to database and server backups.

To restore a Stardog database from a Stardog backup volume, simply pass a fully-qualified path to the volume in question. The location of the backup should be the full path to the backup, not the location of the backup directory as specified in your Stardog configuration. There is no need to specify the name of the database to restore.

To restore a database from its backup:

Backups can also be restored directly from S3 by using an S3 URL in the following format:

Note: Unlike the backup URL the database name must be specified as the last entry of the path field in the URL.

Stardog can be configured to automatically restore databases from a backup location on startup. For example, when a Stardog cluster node first starts it could pull all of the database data down from an S3 backup before joining the cluster.

There are two options that control this behavior.

You can use server restore command to restore server backups created by server backup. To do so you must shut down Stardog and set $STARDOG_HOME to an empty home directory. The server restore command will restore the complete server to $STARDOG_HOME. Once complete you can start Stardog server.

Server backups do not contain your license file, stardog.properties or any other additional files or directories created externally under STARDOG_HOME so you need to back up adn restore those files and directories separately.

A file passed to create will be treated as compressed if the file name ends with .gz or .bz2. The RDF format of the file is determined by the penultimate extension. For example, if a file named test.ttl.gz is used as input, Stardog will perform GZIP decompression during loading and parse the file with Turtle parser. All the formats supported by Stardog (RDF/XML, Turtle, Trig, etc.) can be used with compression.

The ZIP support works differently since zipped files can contain many files. When an input file name ends with .zip, Stardog performs ZIP decompression and tries to load all the files inside the ZIP file. The RDF format of the files inside the zip is determined by their file names as usual. If there is an unrecognized file extension (e.g. '.txt'), then that file will be skipped.

Functions are useful to encapsulate computational or business logic for reuse. We can create a new function to compute the permutation using the function add command with stardog-admin on the command line:

We can use this function in a SPARQL query and see that the function is expanded in the query plan:

Function definitions provided to the add command must adhere to the following grammar:

We can use IRIs or prefixed names as function names and include several functions in one add call:

The admin commands cover adding, listing and removing functions. Examples of these commands are shown below:

HTTP APIs are also provided to add, list and remove stored functions:

The contents of the POST request should be a document containing one or more function definitions using the syntax describes above. The GET request by default returns the definitions for all the functions. If the name parameter is specified a definition for the function with that name is returned. Similarly, the DELETE request deletes all the functions by default or deletes a single function if the name parameter is specified.

Stored functions are persisted in the system database. The system database should be backed up properly to avoid loss of functions.

Stored functions are compiled at creation time in a way that guarantees they will work indefinitely, even if other functions are removed or changed in ways that would affect them. For this reason, dependent functions need to be reloaded when their dependencies are changed.

Queries can be stored using the stored add admin command and specifying a unique name for the stored query:

If a file is used to specify the query string without an explicit -n/--name option then the name of the query file is used for the stored query:

By default, stored queries can be executed over any database. But they can be scoped by providing a specific database name with the -d/--database option. Also, by default, only the user who stored the query can access that stored query. Using the --shared flag will allow other users to execute the stored query.

The following example stores a shared query with a custom name that can be executed over only the database myDb:

The JSON attributes which correspond to --shared and -d are shared and database.

Stored query names must be unique for a Stardog instance. Existing stored queries can be replaced using the --overwrite option in the command.

Queries can be updated using the --overwrite option on the stored add admin command and specifying an existing name for a stored query:

Stored queries are saved as RDF statements in the Stardog system database and it is possible to export the RDF representation of the queries:

The same RDF representation can be used to import the stored queries as an alternative way of storing new queries or updating existing stored queries.

In addition to the built-in properties from the system database arbitrary RDF properties can be used for stored queries. The value of these additional annotation properties should be IRIs or literals. Only the values directly linked to the stored query subject in the RDF document will be saved and the triples with a non-stored query subject will be ignored.

Stored queries can be executed using the regular query execution CLI command by passing the name of the stored query:

Other commands like query explain also accept stored query names. They can also be passed instead of query string into HTTP API calls.

To see all the stored queries, use the stored list subcommand:

The results are formatted tabularly:

Users can only see the queries they’ve stored and the queries stored by other users that have been --shared. The --verbose option will show more details about the stored queries.

Stored queries can be removed using the stored remove command:

If you would like to clear all the stored queries then use the -a/--all option:

For many uses cases the default configuration will be sufficient. But you may need to tweak the timeout parameter to be longer or shorter, depending on the hardware, data load, queries, throughput, etc. The default configuration has a server-wide query timeout value of query.timeout, which is inherited by all the databases in the server. You can customize the server-wide timeout value and then set per-database custom values, too. Any database without a custom value inherits the server-wide value. To disable query timeout, set query.timeout to 0. If individual queries need to set their own timeout, this can be done (by passing a timeout parameter over HTTP or using the --timeout flag on the CLI), but only if the query.timeout.override.enabled property is set to true for the database (true is the default).

To see all running queries, use the query list subcommand:

The results are formatted tabularly:

You can see which user owns the query (superuser’s can see all running queries), as well as the elapsed time and the database against which the query is running. The ID column is the key to deleting queries.

To terminate a running query, simply pass its ID to the query kill command:

The output confirms the query kill completing successfully:

For production use, especially when a Stardog database is exposed to arbitrary query input, some of which may not execute in an acceptable time, the automatic query killing feature is useful. It will protect a Stardog Server from queries that consume too many resources.

Once the execution time of a query exceeds the value of query.timeout, the query will be killed automatically.^[6] The client that submitted the query will receive an error message. The value of query.timeout may be overriden by setting a different value (smaller or longer) in database options. To disable, set to query.timeout to 0.

The value of query.timeout is a positive integer concatenated with a letter, interpreted as a time duration: 'h' (for hours), 'm' (for minutes), 's' (for seconds), or 'ms' (for milliseconds). For example, '1h' for 1 hour, '5m' for 5 minutes, '90s' for 90 seconds, and '500ms' for 500 milliseconds.

The default value of query.timeout is five minutes.

To see more detail about query in-flight, use the query status subcommand:

The resulting output includes query metadata, including the query itself:

Stardog does not log slow queries in the default configuration because there isn’t a single value for what counts as a "slow query", which is entirely relative to queries, access patterns, dataset sizes, etc. While slow query logging has minimal overhead, what counts as a slow query in some context may be acceptable in another. See Configuring Stardog Server above for the details.

For HTTP protocol support, see Stardog’s Apiary docs.

For Java, see the Javadocs.

The security model for query management is simple: any user can kill any running query submitted by that user, and a superuser can kill any running query. The same general restriction is applied to query status; you cannot see status for a query that you do not own, and a superuser can see the status of every query.

We will use the following running example to explain query plans in Stardog.

This query returns the names of all people who have authored both a journal article and a paper in a conference proceedings. The query plan used by Stardog (in this example, 4.2.2) to evaluate this query is:

The plan is arranged in an hierarchical, tree-like structure. The nodes, called operators, represent units of data processing during evaluation. They correspond to evaluations of graphs patterns or solution modifiers as defined in SPARQL 1.1 specification. All operators can be regarded as functions which may take some data as input and produce some data as output. All input and output data is represented as streams of solutions, that is, sets of bindings of the form x → value where x is a variable used in the query and value is some RDF term (IRI, blank node, or literal). Examples of operators include scans, joins, filters, unions, etc.

Numbers in square brackets after each node refer to the estimated cardinality of the node, i.e. how many solutions Stardog expects this operator to produce when the query is evaluated. Statistics-based cardinality estimation in Stardog merits a separate blog post, but here are the key points for the purpose of reading query plans:

Stardog generally evaluates query plans according to the bottom-up SPARQL semantics. Leaf nodes are evaluated first and without input, and their results are then sent to their parent nodes up the plan. Typical examples of leaf nodes include scans, i.e. evaluations of triple patterns, evaluations of full-text search predicates, and VALUES operators. They contain all information required to produce output, for example, a triple pattern can be directly evaluated against Stardog indexes. Parent nodes, such as joins, unions, or filters, take solutions as inputs and send their results further towards the root of the tree. The root node in the plan, which is typically one of the solution modifiers, produces the final results of the query which are then encoded and sent to the client.

One tricky part of understanding Stardog query plans is that evaluation of each operator in the plan is context-sensitive, i.e. it depends on what other nodes are in the same plan, maybe in a different sub-tree. In particular, the cardinality estimations, even if assumed accurate, only specify how many solutions the operator is expected to produce when evaluated as the root node of a plan.

However, as it is joined with other parts of the plan, the results can be different. This is because Stardog employs optimizations to reduce the number of solutions produced by a node by pruning those which are incompatible with other solutions with which they will later be joined.

Consider the following basic graph pattern and the corresponding plan:

The pattern matches all documents created by a person named Paul Erdoes. Here the second pattern is selective (only one entity is expected to have the name "Paul Erdoes"). This information is propagated to the other two scans in the plan via merge joins, which allows them to skip scanning large parts of data indexes.

In other words, the node Scan[POSC](?erdoes, rdf:type, foaf:Person) [#433K] will not produce all 433K solutions corresponding to all people in the database and, similarly, Scan[POSC](?document, dc:creator, ?erdoes) [#898K] will not go through all 898K document creators.

Performance problems may arise because of two reasons:

It is important to distinguish the two. In the former case the best way forward is to make the patterns in WHERE more selective. In the latter case, i.e. when the query returns some modest number of results but takes an unacceptably long time to do so, one needs to look at the plan, identify the bottlenecks (most often, pipeline breakers), and reformulate the query or report it to us for further analysis.

Here’s an example of a un-selective query:

The query returns all distinct pairs of authors who published (possibly different) articles in the same journal. It returns more than 18M results from a database of 5M triples. Here’s the plan:

This query requires an expensive join on ?journal which is evident from the plan (it’s a hash join in this case). It produces more than 18M results (Stardog expects 17.7M which is pretty accurate here) that need to be filtered and examined for duplicates. Given all this information from the plan, the only reasonable way to address the problem would be to restrict the criteria, e.g. to particular journals, people, time periods, etc.

If a query is well-formulated and selective, but performance is unsatisfactory, one may look closer at the pipeline breakers, e.g. this part of the query plan:

A reasonable thing to do would be to evaluate the join of ?article rdf:type bench:Article and ?article dc:creator ?person separately, i.e. as a separate queries, to see if the estimation of 391K is reasonably accurate and to get an idea about memory pressure. This is a valuable piece of information for a performance problem report, especially when the data cannot be shared with us. Similar analysis can be done for hash joins.

In addition to pipeline breakers, there could be other clear indicators of performance problems. One of them is the presence of LoopJoin nodes in the plan. Stardog implements the nested loop join algorithm which evaluates the join by going through the Cartesian product of its inputs. This is the slowest join algorithm and it is used only as a last resort. It sometimes, but not always, indicates a problem with the query.

Here’s an example:

The query is similar to an earlier query plan we saw but runs much slower. The plan shows why:

The loop join near the top of the plan computes the Cartesian product of the arguments which produces almost 200B solutions. This is because there is no shared variable between the parts of the query which correspond to authors of articles and conference proceedings papers, respectively. The filter condition ?name = ?name2 cannot be transformed into an equi-join because the semantics of term equality used in filters is different from the solution compatibility semantics used for checking join conditions.

The difference manifests itself in the presence of numerical literals, e.g. "1"^^xsd:integer = "1.0"^^xsd:float, where they are different RDF terms. However, as long as all names in the data are strings, one can re-formulate this query by renaming ?name2 to ?name which would enable Stardog to use a more efficient join algorithm.

The following operators are used in Stardog query plans:

Query hints help Stardog generate optimized query plans. They can be expressed in queries in two ways:

The first approach makes hints transparent to other query processing tools (other than Stardog). The second approach is preferred when using 3rd party tools which do not preserve SPARQL comments. In both cases a hint applies to the scope where it’s used and all nested scopes (unless overridden by the same hint with a different value).

The equality.identity hint expects a comma-separated list of variables. It tells Stardog that these variables will be bound to RDF terms (IRIs, bnodes, or literals) for which equality coincides with identity (i.e. any term is equal only to itself). This is not true for literals of certain numerical datatypes (cf. Operator Mapping). However assuming that the listed variables do not take on values of such datatypes can sometimes lead to faster query plans, for example, because of converting some filters to joins and through value inlining.

Sometimes our query planner can produce sub-optimal join orderings. The group.joins hint introduces an explicit scoping mechanism to help with join order optimization. Patterns in the scope of the hint, given by the enclosing {}, will be joined together before being joined with anything else. This way, you can tell the query planner what you think is the optimal way to join variables.

The push.filters hint controls how the query optimizer pushes filters down the query plan. There are three possible values: default, aggressive, and off. The aggressive option means that the optimizer will push every filter to the deepest operator in the plan which binds variables used in the filter expression. The off option turn the optimization off and each filter will be applied to the top operator in the filter’s graph pattern (in case there’re multiple filters, their order is not specified). Finally, the default option (or absence of the hint) means that the optimizer will decide whether to push each filter down the plan based on various factors, e.g. the filter’s cost, selectivity of the graph pattern, etc.

Databases may guarantee atomicity—​groups of database actions (i.e., mutations) are irreducible and indivisible: either all the changes happen or none of them happens. Stardog’s transacted writes are atomic. Stardog does not support nested transactions.^[8]

Data stored should be valid according to the data model (in this case, RDF) and to the guarantees offered by the database, as well as to any application-specific integrity constraints that may exist. Stardog’s transactions are guaranteed not to violate integrity constraints during execution. A transaction that would leave a database in an inconsistent or invalid state is aborted.

See the Validating Constraints section for a more detailed consideration of Stardog’s integrity constraint mechanism.

A Stardog connection will run in SNAPSHOT isolation level if it has not started an explicit transaction and will run in SNAPSHOT or SERIALIZABLE isolation level depending on the value of the transaction.isolation. In any of these modes, uncommitted changes will only be visible to the connection that made the changes: no other connection can see those values before they are committed. Thus, "dirty reads" can never occur. Additionally, a transaction will only see changes which were committed before the transaction began, so there are no "non-repeatable reads".

SNAPSHOT isolation does suffer from the write skew anomaly, which poses a problem when operating under external logical constraints. We illustrate this with the following example, where the database initially has two triples :a :val 100 and :b :val 50, and the application imposes the constraint that the total can never be less than 0.

At the end of this scenario, Connection 1 believes the state of the database to be :a :val 0 and :b :val 50, so the constraint is not violated. Similarly, Connection 2 believes the state of the database to be :a :val 100 and :b :val 0, which also does not violate the constraint. However, Connection 3 sees :a :val 0 and :b :val 0 which violates the logical constraint.

No locks are taken, or any conflict resolution performed, for concurrent transactions in SNAPSHOT isolation level. If there are conflicting changes, the transaction with the highest commit timestamp (functionally, the transaction which committed "last") will be the result held in the database. This may yield unexpected results since every transaction reads from a snapshot that was created at the time its transaction started.

Consider the following query being executed by two concurrent threads in READ COMMITTED SNAPSHOT isolation level against a database having the triple :counter :val 1 initially:

Since each transaction will read the current value from its snapshot, it is possible that both transactions will read the value 1 and insert the value 2 even though we expect the final value to be 3.

Isolation level SERIALIZABLE can be used to avoid these situations. In SERIALIZABLE mode an exclusive lock needs to be acquired before a transaction begins. This ensures concurrent updates cannot interfere with each other, but as a result update throughput will decrease since only one transaction can run at a time.

By default Stardog’s transacted writes are durable and no other actions are required.

Stardog’s transaction framework is low maintenance; but there are some rare conditions in which manual intervention may be needed.

Stardog’s strategy for recovering automatically from commit failure is as follows:

With an appropriate logging configuration for production usage (at least error-level logging), log messages for the preceding recovery operations will occur. If for whatever reason the database fails to be returned automatically to online status, an administrator may use the CLI tools (i.e., stardog-admin db online) to try to online the database.

To create an Elasticsearch virtual graph, you need to download the Elasticsearch client jar along with two supporting jars. The client jar for Elasticsearch version "x.y.z" can be obtained from https://repo1.maven.org/maven2/org/elasticsearch/client/elasticsearch-rest-client/x.y.z/elasticsearch-rest-client-x.y.z.jar

Two supporting jars are also required:

Then follow the instructions here.

To create a Cassandra virtual graph, you’ll need to first download the shaded Cassandra client jar. The client jar for Cassandra version "x.y.z" can be obtained from http://central.maven.org/maven2/com/datastax/cassandra/cassandra-driver-core/x.y.z/cassandra-driver-core-x.y.z-shaded.jar Then follow the instructions here.

Cassandra is special in the way it attempts to prevent users from distributing queries over a large number of server nodes. If you have experience with CQL queries, you have no doubt seen the ubiquitous error message, Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING.

This reflects the Cassandra modeling principle that favors writing the same data to multiple tables (perhaps through the use of Materialized Views), where each table is optimized for answering different queries.

In order to support as many queries as possible, we recommend creating mappings to each of these tables and letting Stardog choose which mappings apply for each query. It is possible that no mappings can support a particular query. In such cases, Stardog will write an entry to the log file and return no results.

This is the default behavior, which can be changed by setting the cassandra.allow.filtering virtual graph option to true. When set, Stardog will include the ALLOW FILTERING clause at the end of each CQL query. Please note that the use of this option is highly discouraged in large-scale production environments.

Cassandra is also special for how SQL-like its query language is (for a NoSQL database). As this is the case, Stardog supports the use of SQL queries in the mappings files for Cassandra virtual graphs. That is, you can use the rr:sqlQuery predicate for R2RML mappings, the sm:query predicate for Stardog Mapping Syntax, or the FROM SQL clause for Stardog Mapping Syntax 2. In all cases, you can supply a SQL query to describe a view to use for a virtual graph mapping, however, the SQL query can only contain operators that are supported in CQL - no joins, subqueries, SQL functions, etc. are allowed.

To import a CSV file, provide the file as the last argument to the import command:

If the input file is using different kind of separators, e.g. tab character, a properties file can be provided:

The properties file for CSVs can specify values for the following properties:

The csv.escape character is used as an alternative to the csv.quote character. To escape a csv.quote character within a string that is enclosed in csv.quote characters, use two consecutive csv.quote characters. Do not set csv.escape to the csv.quote character.

Note that whitespace characters in the Java properties file need to be escaped so if you want to import tab-separated value files set csv.separator=\t in the properties file.

In addition to directly referencing columns by name, CSV mappings may include a special ?ROW_NUMBER variable used to obtain the current line number.

The mappings for delimited files can be automatically generated given a couple additional properties:

To import with automatically generated mappings, omit the command line argument for the mappings file:

There is a complete example available in our examples repo.

To import a JSON file, provide the file name as the final argument to the import command:

Here is an example JSON file:

and a corresponding SMS2 mapping:

Note how unlike from json when used with MongoDB, with a JSON file SMS2 mapping there is no MongoDB collection name serving as the key name for the top-level object.

To connect to your data sources, Stardog requires you to supply Stardog with the appropriate Client Driver. You need to manually copy the JAR file containing the driver to the stardog_install_dir/server/dbms/ directory (or to the location pointed to by the STARDOG_EXT environment variable if you have set one) so that it will be available to the Stardog server.

Stardog is tested against all supported databases using the following drivers. Stardog requires a JDBC 4.2 compatible client driver. While other drivers may work, your mileage may vary. For best results, set the sql.dialect option property when using unsupported drivers.

The following table lists the available options for use in virtual graph properties files. The first prefix indicates the type of datasource that the property is used for. jdbc. properties are used for all relational data sources.

Additionally, connection pool properties for the built-in Tomcat connection pool are allowed. This set of additional allowed properties is listed in the Tomcat JDBC Connection Pool documentation. Stardog sets these connection pool defaults: initialSize=3, testWhileIdle=true, timeBetweenEvictionRunsMillis=14400000 (4 hours), and validationQueryTimeout=10.

Any options with the prefix ext. will be passed directly to the JDBC Driver.

Any unknown options will be ignored.

R2RML is the W3C-recommended language for mapping relational databases to RDF. For this reason all SMS mappings, and all SMS2 mappings using FROM SQL, can be converted to R2RML.

The Stardog Mapping Syntax (SMS) is an alternative way to write R2RML mappings that is much simpler to read and write than R2RML.

We will use the example database from the R2RML specification to explain SMS. The SQL schema that corresponds to this example is:

Suppose we would like to represent this information in RDF using the same translation for job codes as in the original example:

SMS looks very similar to the output RDF representation:

SMS is based on Turtle, but it’s not valid Turtle since it uses the URI templates of R2RML—​curly braces can appear in URIs. Other than this difference, we can treat an SMS document as a set of RDF triples. SMS documents use the special namespace tag:stardog:api:mapping: that we will represent with the sm prefix below.

Every subject in the SMS document that has a sm:map property maps a single row from the corresponding table/view to one or more triples. If an existing table/view is being mapped, sm:table is used to refer to the table. Alternatively, a SQL query can be provided inline using the sm:query property.

The values generated will be a URI, blank node, or a literal based on the type of the value used in the mapping. The column names referenced between curly braces will be replaced with the corresponding values from the matching row.

SMS can be translated to the standard R2RML syntax automatically by Stardog. For completeness, we provide the R2RML mappings corresponding to the above example:

Stardog Mapping Syntax 2 (SMS2) is a way to represent virtual graph mappings that is designed to support a broader range of source data formats than R2RML and SMS, including semi-structured data sources such as JSON, MongoDB and Elasticsearch, as well as structured formats like SQL RDBMS.

SMS2 is loosely based on the SPARQL CONSTRUCT query. An abbreviated example looks like this:

SMS2 consists of five parts: PROLOGUE, MAPPING, FROM, TO, and WHERE.

Notice there are no platform-specific query elements (such as MongoDB query syntax) present in the mapping, only descriptions of the source and target data schemas and transformations for mapping the relationship between the source and target.

To help illustrate SMS2, we’ll use the following JSON for a movie collection from a MongoDB database:

For this example we’ll create mappings that represent the data as this RDF:

Notice there are many IRIs that contain both Movie and Person ids. These scoped IRIs are redundant in this dataset but they serve a purpose when working with denormalized datasources, which is common in NoSQL databases like MongoDB. In this dataset, the name of a Person can appear in both actor and director objects. The name is repeated for every directing or acting job that Person has had. There is no guarantee that a Person’s name is constant across all their jobs, either because the field reflects the name the person had at the time of the job, or because of a problem during an update that led to the inconsistency. Without IRIs that scope a Person to a specific Movie, when you query for the Person’s name, the correct response is a record for every Person/name pair, which can be an expensive query. See the blog post Mapping Denormalized Data for more details.

Here is the SMS2 mapping for this exercise:

Details of the various FROM formats follows.

To query a non-materialized Virtual Graph it must first be registered with Stardog. Adding a new virtual graph is done via the following command:

When adding a Virtual Graph Stardog will establish a connection to the data source to verify the provided configuration and mappings.

Properties file

The properties file (dept.properties in this example) contains all of the configuration for the JDBC data source and virtual graph configuration. It must be in the Java properties file format.

A minimal example (in this case, for MySQL) looks like this:

The credentials for the JDBC connection need to be provided in plain text. An alternative way to provide credentials is to use the password file mechanism. The credentials should be stored in a password file called services.sdpass located in STARDOG_HOME directory. The password file entries are in the format hostname:port:database:username:password so for the above example there should be an entry localhost:*:dept:dept:MySqlUserName:MyPassword in this file. Then the credentials in the properties file can be omitted.

The properties file can also contain a property called base to specify a base URI for resolving relative URIs generated by the mappings (if any). If no value is provided, the base URI will be virtual://myGraph where myGraph is the name of the virtual graph.

Mapping file

The mapping file (dept.ttl in this example) contains the mapping from the virtual data source into RDF. The mapping can be in one of three formats:

A mapping file is required for data sources without a built-in schema, e.g. some NoSQL databases like MongoDB.

A mapping file is not required if your data has a built-in schema, e.g. MySQL or other relational databases. In this case you can omit a mapping file and the the virtual graph will be automatically mapped using R2RML direct mapping. Omitting a mapping file is most commonly used with one or both of the options default.mapping.include.tables and sql.schemas to indicate the specific tables to include.

Registered virtual graphs can be listed:

Notice the * in the Database column of the output of the ` virtual list` command. This indicates that the dept virtual graph can be used with any database. To associate a virtual graph with a specific database, use the -d <db> or --database <db> command-line option with the virtual add command.

If a virtual graph fails to load during startup it will be listed as offline (Online false). Use the virtual online command to retry loading an offline virtual graph.

The commands virtual mappings and virtual options can be used to retrieve the mappings and configuration options associated with a virtual graph, respectively. Registered virtual graphs can be removed using the virtual remove command. See the Man Pages for the details of these commands.

Querying Virtual Graphs is done by using the GRAPH clause, using a special graph URI in the form virtual://myGraph to query the Virtual Graph named myGraph.

The following example shows how to query dept:

Virtual graphs can be defined globally in Stardog Server, which is the default, or they can be linked to a specific database when they are created (see here). If a virtual graph is linked to a specific database, it can only be accessed from that database. Attempts to access a linked virtual graph from some other database will result in no data being returned from that virtual graph.

Once a virtual graph is registered, it can be accessed as allowed by the access rules.

We can query the local Stardog database and virtual graph’s remote data in a single query. Suppose we have the dept virtual graph, defined as above, that contains employee and department information, and the Stardog database contains data about the interests of people. We can use the following query to combine the information from both sources:

Or, with Virtual Transparency enabled, the following query will include remote data from the virtual graph as well as from the default graph.

Virtual Graph queries are implemented by executing a query against the remote data source. This is a powerful feature and care must be taken to ensure peak performance. SPARQL and SQL don’t have feature parity, especially given the varying capabilities of SQL implementations. Stardog’s query translator supports most of the salient features of SPARQL including:

That said, there are also limitations on translated queries. This includes:

In some cases you need to materialize the information stored in RDBMS directly into RDF. For example, a combination of high network latency, slow-changing data, and strict query performance requirements can make materialization a good fit.

There is a command virtual import that can be used to import the contents of the RDBMS into Stardog. The command can be used as follows:

This command adds all the mapped triples from the RDBMS into the default graph. Similar to virtual add, this command assumes Stardog Mapping Syntax by default and can accept R2RML mappings using the --format r2rml option or Stardog Mapping Syntax 2 mappings using the --format sms2 option.

It is also possible to specify a target named graph by using the -g/--namedGraph option:

This virtual import command is equivalent to the following SPARQL update query:

If the RDBMS contents change over time, and we need to update the materialization results in the future, we can clear the named graph contents and rematerialize again. This can be done by using the --remove-all option in virtual import or with the following SPARQL query:

Query performance over materialized graphs will be better as the data will be indexed locally by Stardog, but materialization may not be practical in cases where frequency of change is very high.

To manage virtual graphs, the user must be granted access to the virtual-graph security resource type (see Security). create permission is required to add a virtual graph, delete permission is needed to either remove or add a virtual graph with the -o or --overwrite option, and read permission is required for all other management commands such as options or mappings.

Accessing virtual graphs is controlled the same way as regular named graphs as explained in the Named Graph Security section. If named graph security is not enabled for a database, all registered virtual graphs in the server will be accessible through that database. If named graph security is enabled for a database, then users will be able to query only the virtual graphs for which they have been granted access.

If the virtual graphs contain any sensitive information, then it is recommended to enable named graph security globally by setting security.named.graphs=true in stardog.properties. Otherwise creating a new database without proper configuration would allow users to access those virtual graphs.

The Named Graph Security settings apply to virtual graphs regardless of the manner in which they are accessed. The following three queries are identical with the one exception that attempts to access a virtual graph using the SERVICE keyword result in an error when there are insufficient permissions while queries that use the GRAPH or FROM keywords will treat the virtual graphs as empty and return no results but without error.

Stardog uses log4j2 for logging. You can enable debug logging at the virtual graph level by adding this line to the Loggers section of your log4j2.xml file:

This will increase the logging to the stardog.log file (but not to the console).

You can use the metadata inspection tool to retrive the schema and column names as they appear to Stardog when connecting using the the virtual graph connection properties (driver name, connection string, credentials, etc.). The inspection tool is accessed using the command-line interface (CLI). The documentation is accessed using the help command:

The entities extractor detects the mentions of named entities based on the configured models and creates RDF statements for those entities. When we are putting a document we need to specify that we want to use a non-default extractor. We can use both the tika metadata extractor and the entities extractor at the same time:

The result of entity extraction will be in a named graph where an auto-generated IRI is used for the entity:

The linker extractor performs the same task as entities but after the entities are extracted it links those entities to the existing resources in the database. Linking is done by matching the mention text with the identifier and labels of existing resources in the database. This extractor requires the search feature to be enabled to find the matching candidates and uses string similarity metrics to choose the best match. The commonly used properties for labels are supported: rdfs:label, foaf:name, dc:title, skos:prefLabel and skos:altLabel.

The extraction results of linker will be similar to entities, but only contain existing resources for which a link was found. The link is available through the dc:references property.

The dictionary extractor fullfills the same purpose as the linker, but instead of heuristically trying to match a mention’s text with existent resources, it uses a user-defined dictionary to perform that task. The dictionary provides a set of mappings between text and IRIs. Each mention found in the document will be searched in the dictionary and, if found, the IRIs will be added as dc:references links.

Dictionaries are .linker files, which need to be available in the docs.opennlp.models.path folder. Stardog provides several dictionaries created from Wikipedia and DBPedia, which allow users to automatically link entity mentions to IRIs in those knowledge bases.

When using the dictionary option, all .linker files in the docs.opennlp.models.path folder will be used. The output follows the same syntax as the linker.

User-defined dictionaries can be created programmatically. For example, the Java class below will create a dictionary that links every mention of Tom Hanks to two IRIs.

Both entities, linker, and dictionary extractors are also available as a SPARQL service, which makes them applicable to any data in the graph, whether stored directly in Stardog or accessed remotely on SPARQL endpoints or virtual graphs.

The entities extractor is accessed by using the docs:entityExtractor service, which receives one input argument, docs:text, with the text to be analyzed. The output will be the extracted named entity mentions, bound to the variable given in the docs:mention property.

By adding an extra output variable, docs:entity, the linker extractor will be used instead.

The dictionary extractor is called in a similar way to linker, with an extra argument docs:mode set to docs:Dictionary.

All extractors accept one more output variable, docs:type, which will output the type of entity (e.g., Person, Organization), when available.

In order to use stardog-graviton in its current form the following environment variables must be set.

The account associated with the access tokens must have the ability to create IAM credentials and full EC2 access.

Both terraform and packer must be in your system path.

The easiest way to launch a cluster is to run stardog-graviton in interactive mode. This will cause the program to ask a series of questions in order to get the needed values to launch a cluster. Here is a sample session:

To avoid being asked questions a file named ~/.graviton/default.json can be created. An example can be found in the defaults.json.example file.

All of the components needed to run a Stardog cluster are considered part of a deployment. Every deployment must be given a name that is unique to each cloud account. In the above example the deployment name is mystardog2.

Once the image has been successfully launched its health can be monitored with the status command:

AWS EC2 charges by the hour for the VMs that Graviton runs thus when the cluster is no longer in use it is important to clean it up with the destroy command.

For more information about Graviton check out the README and the blog post.

Similar to single-node Stardog, you can restore individual databases with db restore and the cluster will replicate the database to all nodes in the cluster.

Because Stardog Cluster uses ZooKeeper to ensure strong consistency between all of the nodes in the cluster, we recommend server restore be done on only one node with a fresh deploy of ZooKeeper (i.e., clear ZooKeeper’s state once it is no longer in use).

The operational process is to: