TOSCA

Introduction 

The Topology and Orchestration Specification for Cloud Applications (TOSCA) is an OASIS open standard that provides language to describe a topology of cloud based web services, their components, relationships, and the processes that manage them. TOSCA provides mechanisms for abstraction and composition, thereby enabling portability and automated management across cloud providers regardless of underlying platform or infrastructure.

The TOSCA specification allows the user to define a Service Template which describes an online application or service and how to deploy it.

Core TOSCA Concepts 

The figure below illustrates the core components of a TOSCA service template. At the heart of a TOSCA service template is a Topology Template that describes a model of the service’s online resources, including how they connect together and how they should be deployed. These resources are represented by a Node Template. The Node Template and a Relationship Template are the building blocks of any TOSCA Specification.

Node templates describe the resources that will be instantiated when the service template is deployed. They are linked to other nodes through relationships.
Relationship templates can be used to provide additional information about those relationships, for example how to establish a connection between two nodes.
Interfaces are a collections of user-defined operations that are invoked by a TOSCA orchestrator like Unfurl. TOSCA defines a standard interface for lifecycle management operations (creating, starting, stopping and destroying resources) and the user can define additional interfaces for “Day Two” operations, such as maintenance tasks.
Artifacts such as container images, software packages, or files that need to be deployed or used as an implementation for an operation.
Workflows allows you to define a set of manually defined operations to run in a sequential order.
Type definitions TOSCA provides an object-oriented type system that lets you declare types for all of the above components as well as custom data types that provide validation for properties and parameters.

Service Template 

A TOSCA service template contains all the information needed to deploy the service it describes. In Unfurl, a service template can be a stand-alone YAML file that is included in the ensemble.yaml configuration file or embedded directly in that file as a child of the Service templates element.

A service template has the following sections:

Metadata sections, which includes the tosca_definitions_version, description, metadata, dsl_definitions
Imports and Repositories sections
Types sections that contain types of Node, Relationships, Capabilities, Artifacts, Interfaces, Policy and Groups
Topology Template which include sections for Inputs, Outputs, Node and relationship templates, Substitution Mappings, Groups, policies and Workflows.

Example 

tosca_definitions_version: tosca_simple_unfurl_1_0_0 # or use the standard tosca_simple_yaml_1_3
description: An illustrative TOSCA service template
metadata: # the following metadata keys are defined in the TOSCA specification:
  template_name: hello world
  template_author: onecommons
  template_version: 1.0.0

repositories:
   tosca-community-contributions:
     url: https://github.com/oasis-open/tosca-community-contributions.git

 imports:

 - file: my-shared-types.yaml
   namespace_prefix: base # optional
 - file: profiles/orchestration/1.0/profile.yaml
   repository: tosca-community-contributions

node_types:
   # ... see the "types” section below

topology_template:
   # ... see the “topology_templates” section below
   node_templates:
   # ... see the node_templates section below
   relationship_templates:
   # ... see the relationship_templates section below

Types 

Every entity in TOSCA (including Nodes, Relationships, Artifacts and Data) has a declared type. custom type hierarchies can be defined in the Service Template. Types declare the required properties, default definitions, and interface operations for an entity. Each type of entity has can have its own section in the service template, for example, node_types, relationship_types, data_types, artifact_types, interface_types, etc. but type names all are share one namespace in the YAML document or Python module they are defined in.

Example 

This example defines a node type named “MyApplication” that inherits from the “tosca.nodes.SoftwareComponent” node type.

YAML

tosca_definitions_version: tosca_simple_unfurl_1_0_0
node_types:
  MyApplication:
    derived_from: tosca.nodes.SoftwareComponent
    attributes:
      private_address:
        type: string
    properties:
      domain:
        type: string
        default: { get_input: domain }
      ports:
        type: tosca.datatypes.network.PortSpec
    requirements:
      - host:
          capability: tosca.capabilities.Compute
          relationship: tosca.relationships.HostedOn
      - db:
          capability: capabilities.postgresdb
          relationship: tosca.relationships.ConnectsTo
    interfaces:
      # TOSCA defines Standard interface for lifecycle management but you can define your own too
      Standard:
        create: create.sh
        configure: configure.sh
        delete: delete.sh

PYTHON

import unfurl
from typing import Any, Sequence
import tosca
from tosca import Attribute
import unfurl.configurators.shell
from unfurl.tosca_plugins.expr import get_input

class MyApplication(tosca.nodes.SoftwareComponent):
    domain: str = get_input("domain")
    ports: "tosca.datatypes.NetworkPortSpec"

    private_address: str = Attribute()

    host: Sequence[
        "tosca.relationships.HostedOn | tosca.nodes.Compute | tosca.capabilities.Compute"
    ] = ()
    db: "tosca.relationships.ConnectsTo | capabilities_postgresdb"

    def create(self, **kw):
        return unfurl.configurators.shell.ShellConfigurator(
            command=["create.sh"],
        )

    def configure(self, **kw):
        return unfurl.configurators.shell.ShellConfigurator(
            command=["configure.sh"],
        )

    def delete(self, **kw):
        return unfurl.configurators.shell.ShellConfigurator(
            command=["delete.sh"],
        )

Topology Template 

The topology Template defines the components of the service being deployed. It can be thought of as a graph of node templates and other components along with their relationships and dependencies.

Topologies can parameterized with Inputs, define outputs, and contains Node Templates, Relationship Templates, Groups, policies, and Workflows.

Topologies can be embedded in other topologies via Substitution Mappings.

Example 

YAML

topology_template:

  inputs:
    domain:
      type: string
      default: example.com

  outputs:
    url:
      type: string
      # value: { concat: [ https://, { get_input: domain }, ':',  { get_attribute: [ myapp, portspec, source ] }, '/api/events'] }
      # # Unfurl also support ansible-enhanced jinja2 template so you could write this instead:
      value: https://{{ TOPOLOGY.inputs.domain }}:{{ NODES.myApp.portspec.source }}/api/events

PYTHON

from tosca import Eval, TopologyInputs, TopologyOutputs


class Inputs(TopologyInputs):
    domain: str = "example.com"


class Outputs(TopologyOutputs):
    url: str = Eval(
        "https://{{ TOPOLOGY.inputs.domain }}:{{ NODES.myApp.portspec.source }}/api/events"
    )

Deployment blueprints 

A deployment blueprint is an Unfurl extension that allows you to define blueprint that is only applied when its criteria matches the deployment environment. It inherits from the main topology template and node templates with matching names replace the ones in the topology template’s. For example, see this example.

Node Template 

A Node Template defines a node that gets instantiated into an instance (or resource) when the service template is deployed. A node template uses its node type as a prototype and can define properties, capabilities, artifacts, and requirements specific to the template.

Example 

YAML

topology_template:
 node_templates:
  myApp:
    type: MyApplication
    properties:
      ports: 
        eval:
          portspec: "80:8080"
    artifacts:
      image:
        type: tosca.artifacts.Deployment.Image.Container.Docker
        file: myapp:latest
        repository: docker_hub
    requirements:
      - host: compute
      - db:
          node: mydb

  mydb:
    type: PostgresDB
    properties:
       name: mydb

  compute:
    type: tosca.nodes.Compute
    capabilities:
      host:
        properties:
          num_cpus: 1
          disk_size: 200GB
          mem_size: 512MB

PYTHON

import tosca
from tosca import GB, MB

compute = tosca.nodes.Compute(
    host=tosca.capabilities.Compute(
        num_cpus=1,
        disk_size=200 * GB,
        mem_size=512 * MB,
    ),
)

mydb = PostgresDB(name="mydb")

myApp = MyApplication(
    ports=tosca.datatypes.NetworkPortSpec("80:8080"),
    host=[compute],
    db=mydb,
)
myApp.image = tosca.artifacts.DeploymentImageContainerDocker(
    "image",
    file="myapp:latest",
    repository="docker_hub",
)

Requirements and Relationships 

Requirements let you define relationships between nodes and must be assigned a relationship type. Requirements are declared as a list on the node template, allowing multiple requirements with the same name, whose number is constrained by the occurrences keyword in the requirement’s definition.

Relationship types can have properties and have interfaces and operations associated with them. For example, the orchestrator will use operations defined for the built-in Configure interface to create or remove the relationship.

Requirements declared on a node type define the constraints for that requirement, for example the type of node it can target or the number of target matches each node template can have for that requirement.

Requirements declared on a node template define the node that the requirement is targeting, either directly by naming that node template, or by defining the search criteria for finding the match, for example by including a node_filter.

So when the node field refers to an node type in requirement defined on a node template its defining a search criteria for finding a matching node template, but when the requirement is on node type, it is only defining a validation constraint where the node template’s match must implement that type – it won’t use that field to search for a match.

Relationship templates can be declared to provide specific information about the requirement’s relationship with its target node. Relationship templates can be defined directly inline in the requirement or they can be declared separately in the relationship_templates section of the topology and referenced by name.

Examples 

This simple example using built-in TOSCA types that defines the “host” requirement as using the tosca.relationships.hostedOn relationship, so we can define the requirement with just the name of the target node template:

topology_template:
  node_templates:
    myApp:
      type: tosca.nodes.SoftwareComponent
      requirements:
        - host: another_node

The following more complex example first defines a “MyApplication” node type that requires one “db” relationship of type “DatabaseConnection”. Then it defines a node template (“myApp”) with that node type with a “db” requirement that uses a node filter to find the matching target node and a named relationship template of type “DatabaseConnection” with properties for connecting to the database.

YAML

node_types:
    MyApplication:
      derived_from: tosca.nodes.SoftwareComponent
      requirements:
      - db:
          node: tosca.nodes.DBMS
          relationship: DatabaseConnection
          occurrences: [1, 1]

relationship_types:
  DatabaseConnection:
    derived_from: tosca.relationships.ConnectsTo
    properties:
      username:
        type: string
      password:
        type: string
        metadata:
          sensitive: true

topology_template:

  relationship_templates:
    mydb_connection:
      type: DatabaseConnection
      properties:
        username: myapp
        password:
          eval:
            secret:
              myapp_db_pw

  node_templates:
    myApp:
      type: MyApplication
      requirements:
        - db:
            relationship: mydb_connection
            node_filter:
              properties:
                - name: mydb

PYTHON

import tosca
from tosca import Eval, Node, Property, Relationship

class DatabaseConnection(tosca.relationships.ConnectsTo):
    username: str
    password: str = Property(metadata={"sensitive": True})


class MyApplication(tosca.nodes.SoftwareComponent):
    db: "DatabaseConnection | tosca.nodes.DBMS"


mydb_connection = DatabaseConnection(
    "mydb_connection",
    username="myapp",
    password=Eval({"eval": {"secret": "myapp_db_pw"}}),
)
myApp = MyApplication(
    "myApp",
    db=mydb_connection,
)

Artifacts 

An artifact is an entity that represents an asset such as files, container images, or software packages that need to be deployed or used by an implementation of an operation. Like other TOSCA entities, artifacts have a type and can have properties, attributes, and operations associated with it.

Artifacts are declared in the artifacts section of node templates and node types, and can refer to the repository that it is part of.

This example defines an artifact that is a container image, along with a repository that represents the image registry that manages it:

 docker_hub:
   url: https://registry.hub.docker.com/
   credential:
      user: user1
      token:
        eval:
          secret:
            dockerhub_user1_pw

topology_template:
  node_templates:
    myApp:
      type: MyApplication
      artifacts:
        image:
          type: tosca.artifacts.Deployment.Image.Container.Docker
          file: myapp:latest
          repository: docker_hub

Artifacts can be used in the following ways:

An operation’s implementation's, primary field can be assigned an artifact which will be used to execute the operation. Artifacts derived from unfurl.artifacts.HasConfigurator will use configurator set on its type, otherwise it will treated as a shell script unless the className field is set in the implementation.
An implementation can also list artifacts in the dependencies field which will be installed if necessary.
The get_artifact TOSCA function to reference to artifact’s URL or local location (if available).
An artifact and its properties can be accessed in Eval Expressions via the .artifacts key. (see Artifact enhancements) or as Node attributes when using the Python DSL.

Artifacts that are referenced in an operation’s implementation will be installed on the operation’s operation_host (by default, where Unfurl is running) as part of the Job Lifecycle if the artifact has Standard operations (create or configure) defined for it.

Interfaces and Operations 

An Operations defines an invocation on an artifact or configurator. A TOSCA orchestrator like Unfurl instantiates a service template by executing operations. An operation has an Implementation, inputs, and outputs.

Conceptually an operation is comparable to method in an object-oriented language where the implementation is the object upon which the method is invoked, the inputs are the method’s arguments, and its outputs are the method’s return values.

An Interface is a collections of operations that can be defined in the interfaces section on TOSCA types or directly on TOSCA templates for nodes, relationships, and artifacts.

Example 

interfaces:
  Standard:  # this is a built-in interface type so the "type" field is not required
    inputs:
      # inputs defined at this level will be made available to all operations
      foo: bar
    operations:
      # in the simplest case an operation can just be the name of an artifact or shell command
      create: ./myscript.sh
      # a more complex operation:
      configure:
        implementation:
          primary: my_ansible_playbook
          dependencies:
              # unfurl will install artifacts listed as dependencies if they are missing
            - a_ansible_collection
          # Unfurl extensions:
          className: Ansible # the configurator class to use (redundant in this example)
          environment: # environment variables to set when executing the operation
             AN_ENV_VAR: 1
        inputs:
          foo: baz  # overrides the above definition of "foo"
        outputs:
          an_ansible_fact: an_attribute
    # an Unfurl extension for specifying the connections that the operations need to function:
    requirements:
      - unfurl.relationships.ConnectsTo.GoogleCloudProject

Operations can be invoked in the following ways:

The built-in Standard and Configure interfaces are invoked during deploy and undeploy workflows
Unfurl’s built-in Install are invoked for check and discovery.
Custom operations can be invoked with run workflow.
Operations can be invoked directly by custom Workflows, the Delegate configurator, or using Unfurl’s Python apis.

Implementation 

As the example above illustrates, an operation’s implementation field describes which artifacts or configurator (Unfurl’s plugin system for implementing operations) to use to execute the operation.

As an Unfurl extension, an implementation section can also include an environment key with Environment Variables directives and a className field to explicitly name the configurator.

Inputs 

Inputs are passed to the implementation of the operation. A TOSCA type can (optionally) define the expected inputs on a operation in much like a property definition. In Unfurl, inputs are evaluated lazily as needed by the operation’s configurator. Default inputs can defined directly on the interface and will be made available to all operations in that interface. If inputs are defined on multiple types in a type hierarchy for the same operation, they are merged together along with any inputs for that operation defined on directly on the template.

Outputs 

An operation can define an attribute mapping that specifies how to apply the operation’s outputs. The meaning of keys in the mapping depends on the operation’s configurator, for example, a Ansible fact or a Terraform output. If mapping’s value is a string, it names the attribute on the instance where the output will be saved. If the value is null, no attribute mapping will be made but the output will be available to the resultTemplate and saved in the ensemble.

Interface types 

Interface types define names of the operations in an interface along with their inputs and outputs. TOSCA defines a built-in interface types for lifecycle management operations and additional interface types can be declared in the service template for “Day Two” operations, such as maintenance tasks.

For example, Unfurl adds a built-in interface type for discovering resources, which it defines as:

interface_types:
  unfurl.interfaces.Install:
    derived_from: tosca.interfaces.Root
    operations:
      check:
        description: Checks and sets the status and attributes of the instance
      discover:
        description: Discovers current state of the current instance and (possibly) related instances, updates the spec as needed.
      revert:
        description: Restore the instance to the state it was original found in.
      connect:
        description: Connect to a pre-existing resource.
      restart:
        description: Restart the resource.

Built-in interfaces 

TOSCA defines two built-in interface types that are invoked to deploy a topology: the Standard interface for node templates and Configure interface for relationships templates.

The former defines lifecycle management operations (creating, starting, stopping and destroying resources) that are invoked when creating and deleting node instances and latter for configuring the Requirements between nodes.

It is not a requirement to define every operation for every template; the operation will be silently skipped if not defined. With the Standard interface, only the configure operation needs to be defined to instantiate a node, if the create operation isn’t defined, Unfurl will assume that configure will both create and configure the resource (and reconfigure when updating an existing deployment).

In addition, Unfurl provides a built-in Install interface which is invoked when running check and discover workflows.

Complete Example 

Combining the above examples into one file, we have a complete service template:

YAML

tosca_definitions_version: tosca_simple_unfurl_1_0_0 # or use the standard tosca_simple_yaml_1_3
description: An illustrative TOSCA service template
metadata: # the following metadata keys are defined in the TOSCA specification:
  template_name: hello world
  template_author: onecommons
  template_version: 1.0.0

repositories:
  docker_hub:
    url: https://registry.hub.docker.com/
    credential:
      user: user1
      token:
        eval: # eval is an Unfurl extension
          secret: dockerhub_user1_pw

relationship_types:
  DatabaseConnection:
    derived_from: tosca.relationships.ConnectsTo
    properties:
      username:
        type: string
      password:
        type: string
        metadata:
          sensitive: true

node_types:
  MyApplication:
    derived_from: tosca.nodes.SoftwareComponent
    attributes:
      private_address:
        type: string
    properties:
      domain:
        type: string
        default: { get_input: domain }
        ports:
          type: tosca.datatypes.network.PortSpec
    requirements:
      - host:
          capability: tosca.capabilities.Compute
          relationship: tosca.relationships.HostedOn
      - db:
          relationship: DatabaseConnection
    interfaces:
      # TOSCA defines Standard interface for lifecycle management but you can define your own too
      Standard:
        create: create.sh
        configure: configure.sh
        delete: delete.sh

topology_template:
  inputs:
    domain:
      type: string

  outputs:
    url:
      type: string
      value:
        {
          concat:
            [
              https://,
              { get_input: domain },
              /api/events
            ],
        }
      # Unfurl also support ansible-enhanced jinja2 template so you could write this instead:
      # value: https://{{ TOPOLOGY.inputs.domain }}

  node_templates:
    myApp:
      type: MyApplication
      artifacts:
        image:
          type: tosca.artifacts.Deployment.Image.Container.Docker
          file: myapp:latest
          repository: docker_hub
      requirements:
        - host: compute
        - db:
            node: mydb
            relationship: mydb_connection

    mydb:
      type: tosca.nodes.Database
      properties:
        name: mydb
      requirements:
        - host: compute

    compute:
      type: tosca.nodes.Compute
      capabilities:
        host:
          properties:
            num_cpus: 1
            disk_size: 200GB
            mem_size: 512MB

  relationship_templates:
    mydb_connection:
      type: DatabaseConnection
      properties:
        username: myapp
        password:
          eval:
            secret: myapp_db_pw

PYTHON

# Generated by tosca.yaml2python from docs/examples/service-template.yaml at 2024-07-26T13:42:20 overwrite not modified (change to "overwrite ok" to allow)
"""An illustrative TOSCA service template"""

import unfurl
from typing import Sequence, Union
import tosca
from tosca import Attribute, Eval, GB, MB, Property, TopologyInputs, TopologyOutputs
import unfurl.configurators.shell
from unfurl.tosca_plugins.expr import concat


class Inputs(TopologyInputs):
    domain: str


class Outputs(TopologyOutputs):
    url: str = concat("https://", Inputs.domain, "/api/events")


class DatabaseConnection(tosca.relationships.ConnectsTo):
    username: str
    password: str = Property(metadata={"sensitive": True})


class MyApplication(tosca.nodes.SoftwareComponent):
    domain: str = Inputs.domain

    private_address: str = Attribute()

    host: Sequence[
        "tosca.relationships.HostedOn | tosca.nodes.Compute | tosca.capabilities.Compute"
    ] = ()

    db: "DatabaseConnection"

    def create(self, **kw):
        return unfurl.configurators.shell.ShellConfigurator(
            command=["create.sh"],
        )

    def configure(self, **kw):
        return unfurl.configurators.shell.ShellConfigurator(
            command=["configure.sh"],
        )

    def delete(self, **kw):
        return unfurl.configurators.shell.ShellConfigurator(
            command=["delete.sh"],
        )


compute = tosca.nodes.Compute(
    "compute",
    host=tosca.capabilities.Compute(
        num_cpus=1,
        disk_size=200 * GB,
        mem_size=512 * MB,
    ),
)

mydb = tosca.nodes.Database(
    "mydb",
    name="mydb",
    host=[compute],
)

mydb_connection = DatabaseConnection(
    "mydb_connection",
    username="myapp",
    password=Eval({"eval": {"secret": "myapp_db_pw"}}),
)
myApp = MyApplication(
    "myApp",
    host=[compute],
    db=mydb_connection[mydb],
)
myApp.image = tosca.artifacts.DeploymentImageContainerDocker(
    "image",
    file="myapp:latest",
    repository="docker_hub",
)