Stardog and Pivotal
We ❤️ Pivotal Cloud Foundry and now Stardog runs on it seamlessly.
TLDR: Our integration with Pivotal enables apps running in Pivotal Cloud Foundry to natively connect to Stardog instances without having to manually wire apps to services. Just download the Stardog Service Broker for Pivotal Cloud Foundry and start using Stardog in a few minutes.
With the release of Stardog 5.0 we are proud to announce the Stardog Service Broker for Pivotal Cloud Foundry®. This is a release of the Stardog service broker packaged specifically for Pivotal’s Ops Manager.
While the Stardog service broker is useful for all Cloud Foundry deployments, this packaging makes it particularly convenient for use with Pivotal Cloud Foundry on-prem and public Cloud deployments.
Architecture & Workflow
In this post we will describe the architecture of the Stardog Service Broker for Pivotal Cloud Foundry (SSB) and then walk through everything needed to get it running. In a future post we will discuss how to write apps using this infrastructure.
Cloud Foundry Service Broker Overview
A service broker is the part of Cloud Foundry that connects platform applications to infrastructure services. Infrastructure services are typically things like databases, message queues, storage systems, and—now!—an enterprise knowledge graph.
A service broker does not run the service itself; rather it gathers resources on behalf of its clients. Among other things, its duties include creating services, managing credentials to access those services, tracking the services, and disposing of them when they are no longer needed.
The typical Cloud Foundry service broker workflow is as follows:
Deploy a service broker into Cloud Foundry. This simply starts the service broker process and allows it to start advertising itself as ready for use.
Create a service instance. Resources are created here: a client requests the creation of a new service according to a given plan. A broker may offer many plans where each is a different means of creating a resource. For example, a database service broker may offer two plans, one is pre-provisioned which expects to be given the access information to a running database server as parameters. It uses that information to execute a
CREATE DATABASEinstruction. A second on-demand plan may be offered, as well, which deploys an entire database server into a cloud.
Bind an application to a service instance. In this step a running (or about to be launched) Cloud Foundry application is bound to the service instance created in #2. All the information needed to access the resource associated with the service instance is collected and injected into the environment of the application. From there the application can pull this information out of its environments and connect to the service. In the database example above this step would look up the connection information to the service instance created in #2. It would then create a new user and password on that database. The connection information and new user credentials are then returned to Cloud Foundry where they are injected into the binding applications environment.
Stardog Service Broker
SSB runs as a regular Cloud Foundry application and serves a REST API that adheres to the OpenServiceBroker API
In the first release of SSB, only one plan is offered called perinstance. This plan requires that a pre-provisioned Stardog server is running when a call to create service is made. At this time the service broker will create a new graph database inside of that Stardog server. When an application is bound to the service a new Stardog user is created that has read-write access to the newly created graph database.
Here we will walk through the process of getting a Stardog service instance running in Pivotal Cloud Foundry. The following dependencies are needed:
- A Pivotal Cloud Foundry environment with access to the Ops Manager.
- Stardog Graviton.
- Stardog 5.0 and a license key
- An Amazon Web Services account.
Get The Service Broker Release
First, download the Stardog service broker tile.
Then navigate to your Ops Manager, click on “Import a Product”, and select the file that was downloaded above as shown below.
Once the product is imported it will appear in list of pending products on the left hand side. Under its name is a garbage can icon and a plus icon. Click the plus (circled in red below) to install the tile.
Now that the tile is installed it should be displayed in the list; however, unlike the other tiles in the list it will have an orange bar under it as shown below:
This indicates that the tile is unconfigured. Click on the tile to reveal the configuration screen. In typical Pivotal Cloud Foundry deployments there should be nothing to configure except for selecting a network, which assumes that the p-mysql service broker is installed and that it offers the 100mb plan. Click on the Stardog Metadata Service tab to reveal a screen similar to the one below:
By default the p-mysql service broker and the 100mb plan are selected. If the Pivotal Cloud Foundry deployment in use lacks one or both of those, you will have to enter the names of the Pivotal MySQL service broker and plan that you plan to use.
Pivotal MySQL is used for storing small amount of metadata. It’s possible to use Stardog as the metadata storage device as well, but to make things as simple as possible (and avoid a confusing chicken and egg problem), we chose to only support the MySQL driver in the Pivotal Cloud Foundry tile deployment. When running the Stardog service broker in other environments we recommend using the Stardog storage driver.
Once configured there is one last step that can take some time. On the upper right hand side there is button labeled “Apply Changes” (shown below). Click on this to commit all installation steps to the Ops Manager.
At this point the service broker is installed into the Pivotal Cloud Foundry environment and ready for use.
Launch Stardog with Graviton
Now that Pivotal Cloud Foundry is configured with the Stardog service broker we will need a running Stardog service for it to use. Although any addressable Stardog server will do (cluster or single node), the easiest way to get one running is with Stardog Graviton
While details of Graviton are discussed elsewhere a quick walk-through of everything needed for the launch of a three node cluster is as follows.
Download the latest version of graviton suitable for your architecture. You will also need to have Terraform and Packer in your system path. Graviton will be using AWS to run Stardog therefore the following environment variables must be set:
export AWS_ACCESS_KEY_ID=<a valid aws access key> export AWS_SECRET_ACCESS_KEY=<a valid aws secret key> export STARDOG_ADMIN_PASSWORD=<a password for the admin user>
Once this is all in place simply run
stardog-graviton launch mypcf and
a text wizard will guide you through the rest of the process as shown
$ ./stardog-graviton --log-level=DEBUG launch --cidr="0.0.0.0/0" mystardogpcf What version of stardog are you launching?: 5.0-beta What is the path to the Stardog release?: /usr/local/src/stardog-5.0-beta.zip There is no base image for version 5.0-beta. Do you wish to build one? (yes/no): yes - Running packer to build the image... done AMI Successfully built: ami-ec1c3a8c Creating the new deployment mystardogpcf Would you like to create an SSH key pair? (yes/no): yes What is the path to your Stardog license?: /usr/local/src/stardog-license-key.bin / Calling out to terraform to create the volumes... | Calling out to terraform to stop builder instances... Successfully created the volumes. - Creating the instance VMs...... Successfully created the instance. Waiting for stardog to come up... The instance is healthy Changing the default password... Password changed successfully for user admin. \ Opening the firewall...... \ Opening the firewall...... Successfully opened up the instance. The instance is healthy The instance is healthy Stardog is available here: http://mystardogpcfsdelb-1110206713.us-west-1.elb.amazonaws.com:5821 ssh is available here: mystardogpcfbelb-896006501.us-west-1.elb.amazonaws.com Using 3 stardog nodes 10.0.100.155:5821 10.0.101.158:5821 10.0.100.70:5821 Success.
It can take around 10 minutes for this command to complete.
Note the URL of the newly launched Stardog server in the output. The
administrator user name is
admin and the password is the value set in
then environment variable
STARDOG_ADMIN_PASSWORD above. These three
values will be need later.
Create A Service Instance
Now that the service broker is installed, and Stardog is ready for use, we can create a service instance. Navigate your browser to the your Pivotal Cloud Foundry Apps Manager and log in. Click on the “Marketplace” tab on the left hand side. This will bring up a list of tiles and “Stardog” should be one of them and should look similar to what is below:
Click on that and select the perinstance instance plan (it should be the only one available). From there a screen will appear that will allow you to tell Pivotal Cloud Foundry about the Stardog service that was started with Graviton in the previous step.
Along with a name for this specific instance, three parameters are needed:
- url: The URL of the Stardog server from the graviton output.
- username: admin
- password: The value STARDOG_ADMIN_PASSWORD was set to when graviton launched Stardog.
Once the information is entered click the Add button and a Stardog service instance will be created and ready for use by your Pivotal Cloud Foundry applications.
In a future post we will discuss how Pivotal Cloud Foundry applications can be bound to the service instance that we created here, and how they can then leverage the Stardog Knowledge Graph.
Download Stardog today to start your free 30-day evaluation.
16 May 2017