Stardog and Pivotal

by John Bresnahan, 16 May 2017 · 8 minute read

We ❤️ Pivotal Cloud Foundry and now Stardog runs on it seamlessly.

TLDR: Our integration with Pivotal enables apps running in Pivotal Cloud Foundry to natively connect to Stardog instances without having to manually wire apps to services. Just download the Stardog Service Broker for Pivotal Cloud Foundry and start using Stardog in a few minutes.

Intro

With the release of Stardog 5.0 we are proud to announce the Stardog Service Broker for Pivotal Cloud Foundry®. This is a release of the Stardog service broker packaged specifically for Pivotal’s Ops Manager.

Courtesy of Clinton Steeds

Courtesy of Clinton Steeds

While the Stardog service broker is useful for all Cloud Foundry deployments, this packaging makes it particularly convenient for use with Pivotal Cloud Foundry on-prem and public Cloud deployments.

Architecture & Workflow

In this post we will describe the architecture of the Stardog Service Broker for Pivotal Cloud Foundry (SSB) and then walk through everything needed to get it running. In a future post we will discuss how to write apps using this infrastructure.

Cloud Foundry Service Broker Overview

A service broker is the part of Cloud Foundry that connects platform applications to infrastructure services. Infrastructure services are typically things like databases, message queues, storage systems, and—now!—an enterprise knowledge graph.

A service broker does not run the service itself; rather it gathers resources on behalf of its clients. Among other things, its duties include creating services, managing credentials to access those services, tracking the services, and disposing of them when they are no longer needed.

The typical Cloud Foundry service broker workflow is as follows:

  1. Deploy a service broker into Cloud Foundry. This simply starts the service broker process and allows it to start advertising itself as ready for use.

  2. Create a service instance. Resources are created here: a client requests the creation of a new service according to a given plan. A broker may offer many plans where each is a different means of creating a resource. For example, a database service broker may offer two plans, one is pre-provisioned which expects to be given the access information to a running database server as parameters. It uses that information to execute a CREATE DATABASE instruction. A second on-demand plan may be offered, as well, which deploys an entire database server into a cloud.

  3. Bind an application to a service instance. In this step a running (or about to be launched) Cloud Foundry application is bound to the service instance created in #2. All the information needed to access the resource associated with the service instance is collected and injected into the environment of the application. From there the application can pull this information out of its environments and connect to the service. In the database example above this step would look up the connection information to the service instance created in #2. It would then create a new user and password on that database. The connection information and new user credentials are then returned to Cloud Foundry where they are injected into the binding applications environment.

Courtesy of Frozenatom

Courtesy of Frozenatom

Stardog Service Broker

SSB runs as a regular Cloud Foundry application and serves a REST API that adheres to the OpenServiceBroker API

In the first release of SSB, only one plan is offered called perinstance. This plan requires that a pre-provisioned Stardog server is running when a call to create service is made. At this time the service broker will create a new graph database inside of that Stardog server. When an application is bound to the service a new Stardog user is created that has read-write access to the newly created graph database.

Hands On

Here we will walk through the process of getting a Stardog service instance running in Pivotal Cloud Foundry. The following dependencies are needed:

Get The Service Broker Release

First, download the Stardog service broker tile.

Then navigate to your Ops Manager, click on “Import a Product”, and select the file that was downloaded above as shown below.

Import product

Import product

Once the product is imported it will appear in list of pending products on the left hand side. Under its name is a garbage can icon and a plus icon. Click the plus (circled in red below) to install the tile.

Install the tile

Install the tile

Now that the tile is installed it should be displayed in the list; however, unlike the other tiles in the list it will have an orange bar under it as shown below:

unconfigured tile

unconfigured tile

This indicates that the tile is unconfigured. Click on the tile to reveal the configuration screen. In typical Pivotal Cloud Foundry deployments there should be nothing to configure except for selecting a network, which assumes that the p-mysql service broker is installed and that it offers the 100mb plan. Click on the Stardog Metadata Service tab to reveal a screen similar to the one below:

Configure the storage service tile

Configure the storage service tile

By default the p-mysql service broker and the 100mb plan are selected. If the Pivotal Cloud Foundry deployment in use lacks one or both of those, you will have to enter the names of the Pivotal MySQL service broker and plan that you plan to use.

Pivotal MySQL is used for storing small amount of metadata. It’s possible to use Stardog as the metadata storage device as well, but to make things as simple as possible (and avoid a confusing chicken and egg problem), we chose to only support the MySQL driver in the Pivotal Cloud Foundry tile deployment. When running the Stardog service broker in other environments we recommend using the Stardog storage driver.

Once configured there is one last step that can take some time. On the upper right hand side there is button labeled “Apply Changes” (shown below). Click on this to commit all installation steps to the Ops Manager.

Apply the changes

Apply the changes

At this point the service broker is installed into the Pivotal Cloud Foundry environment and ready for use.

Launch Stardog with Graviton

Now that Pivotal Cloud Foundry is configured with the Stardog service broker we will need a running Stardog service for it to use. Although any addressable Stardog server will do (cluster or single node), the easiest way to get one running is with Stardog Graviton

While details of Graviton are discussed elsewhere a quick walk-through of everything needed for the launch of a three node cluster is as follows.

Download the latest version of graviton suitable for your architecture. You will also need to have Terraform and Packer in your system path. Graviton will be using AWS to run Stardog therefore the following environment variables must be set:

export AWS_ACCESS_KEY_ID=<a valid aws access key>
export AWS_SECRET_ACCESS_KEY=<a valid aws secret key>
export STARDOG_ADMIN_PASSWORD=<a password for the admin user>

Once this is all in place simply run stardog-graviton launch mypcf and a text wizard will guide you through the rest of the process as shown below:

$ ./stardog-graviton --log-level=DEBUG launch --cidr="0.0.0.0/0" mystardogpcf
What version of stardog are you launching?: 5.0-beta
What is the path to the Stardog release?: /usr/local/src/stardog-5.0-beta.zip
There is no base image for version 5.0-beta.
Do you wish to build one? (yes/no): yes
- Running packer to build the image...
done
AMI Successfully built: ami-ec1c3a8c
Creating the new deployment mystardogpcf
Would you like to create an SSH key pair? (yes/no): yes
What is the path to your Stardog license?: /usr/local/src/stardog-license-key.bin
/ Calling out to terraform to create the volumes...
| Calling out to terraform to stop builder instances...
Successfully created the volumes.
- Creating the instance VMs......
Successfully created the instance.
Waiting for stardog to come up...
The instance is healthy
Changing the default password...
Password changed successfully for user admin.
\ Opening the firewall......
\ Opening the firewall......
Successfully opened up the instance.
The instance is healthy
The instance is healthy
Stardog is available here: http://mystardogpcfsdelb-1110206713.us-west-1.elb.amazonaws.com:5821
ssh is available here: mystardogpcfbelb-896006501.us-west-1.elb.amazonaws.com
Using 3 stardog nodes
	10.0.100.155:5821
	10.0.101.158:5821
	10.0.100.70:5821
Success.

It can take around 10 minutes for this command to complete.

Note the URL of the newly launched Stardog server in the output. The administrator user name is admin and the password is the value set in then environment variable STARDOG_ADMIN_PASSWORD above. These three values will be need later.

Create A Service Instance

Now that the service broker is installed, and Stardog is ready for use, we can create a service instance. Navigate your browser to the your Pivotal Cloud Foundry Apps Manager and log in. Click on the “Marketplace” tab on the left hand side. This will bring up a list of tiles and “Stardog” should be one of them and should look similar to what is below:

Select the service

Select the service

Click on that and select the perinstance instance plan (it should be the only one available). From there a screen will appear that will allow you to tell Pivotal Cloud Foundry about the Stardog service that was started with Graviton in the previous step.

Configure the instance

Configure the instance

Along with a name for this specific instance, three parameters are needed:

  1. url: The URL of the Stardog server from the graviton output.
  2. username: admin
  3. password: The value STARDOG_ADMIN_PASSWORD was set to when graviton launched Stardog.

Once the information is entered click the Add button and a Stardog service instance will be created and ready for use by your Pivotal Cloud Foundry applications.

Next

In a future post we will discuss how Pivotal Cloud Foundry applications can be bound to the service instance that we created here, and how they can then leverage the Stardog Knowledge Graph.

Download Stardog today to start your free 30-day evaluation.

John Bresnahan

16 May 2017

Social Media Icon Social Media Icon