Microservices based on yaml files

This tutorial describes how to configure your Microservices and CI server for automatic documentation based on yaml files.

Overview

Pivio.io provides a service to assemble useful information from your source code into yaml files. (Visit: Pivio). This can be

  • name of the system
  • ownership information
  • version control system information (e.g. git)
  • license information
  • software dependencies

Especially, the last one on the list is essential in assessing the risk of using open source software. If you are using open source code in your product and by some piece of code a GPL-like license is introduced, you might be sued into publishing your code - needless to say, your customers might like that but your finance and legal department might not.

1. Quickstart

Put a yaml file into the root of your application. It might look like this:

id: leanix-pathfinder
api_token <api-token from leanix administration page>
name: LeanIX Pathfinder
type: service
owner: LeanIX back-end development
description: Awesome code for providing the back bone of LeanIX EAM application
vcsroot: <your github link>
lifecycle: production
tags:
  - GraphQL
  - Java8
  - back-end
links:
  buildchain: <your CI/CD link>
version: 4.0

The API-token can be acquired from the LeanIX application. Here is a neat way to get it: http://leanix.readme.io/docs/authentication#section-generate-api-tokens

Then, using the docker container leanix/pivio-client from LeanIX, change into your project folder and just execute the command

export PIVIO_SERVICE_URL="https://app.leanix.net/services/integrations/v2/pivio/document"
export PIVIO_API_TOKEN="vL6WOsHFczYDJPqsvs3ExxA6HY7jfSn9VNOKnnQe"
docker run --rm -v $(pwd):/source leanix/pivio-client pivio -serviceurl "${PIVIO_SERVICE_URL}" -verbose -addfield "api_token=${PIVIO_API_TOKEN}"

This will trigger the upload of the yaml file to the LeanIX servers and create your component in LeanIX as ITComponent (of course, you can modify the type of Fact Sheet, if you like) including the information provided.

In case that you have added the field api_token in your yaml file, you don't need to use the argument -addfield.

You can implement your software dependencies manually as well in the yaml file, but we suggest you rather don't and instead use the automatic way as described below.

2. Fully automatic

LeanIX wouldn't be lean, if we didn't provide you with a simple way to integrate the Pivio client with your favourite build system. Currently, the client supports the following build systems

  • NPM
  • Maven

Supporting means, that when starting the modified Pivio client from your source directory, it automatically detects the build system and includes the software dependencies into your yaml file before sending it to the LeanIX servers, which in turn create Fact Sheets for every dependency and link them to the appropriate IT Components. As a plus: License information is added as a tag to your dependency and with LeanIX reports you can easily identify what type of license is used in which source base.

Additionally, most of the time, you don't want to manage attributes of your source-base on multiple points of management, so it is supported to have [autogenerated] in your yaml file on your attributes, if they are instead present in the information of your build system. For example, if your build system is NPM and you have the current version, a description and the name of your application already in your package.json defined, then your YAML could look like this:

id: leanix-pathfinder-web
api_token <api-token from leanix administration page>
name: [autogenerated]
type: service
owner: LeanIX front-end development
description: [autogenerated]
vcsroot: <your github link>
lifecycle: production
tags:
  - Angular
  - JavaScript
  - front-end
  - TypeScript
links:
  buildchain: <your CI/CD link>
version: "[autogenerated]"

2.1 NPM

In order for things to work on your NPM project, you will need to use the npm plugin license-checker which scans the node_modules dependencies our your project. There are two ways to use this plugin:

  • install the license-checker into your project and run there
  • run the license-checker within the docker-container leanix-pivio

2.1.1 Run license-checker directly

Enter your current npm project and install the plugin.

npm install license-checker --save

This will add the license-checker package (and up to 54 more packages associated with it), which enables you to call

./node_modules/license-checker/bin/license-checker --json > dependencies.json

This will store your dependencies in the dependencies.json file.
Then the Pivio client from our docker-container will be able to find those dependencies and merge them with your yaml file and send it to the LeanIX servers.

Name of Dependency File

It is important to name the dependency file dependencies.json. If named differently, the client will not be able to detect the build system correctly.

Ideally, you insert both steps (creating the dependencies.json and the execution of the Pivio client) into your release process, e.g. when your CI/CD system builds a new release from your master branch. That way, you will always have the most up-to-date information available for your management and enterprise architects or simply as overview for your development team.

2.1.2 Run the license-checker inside of a docker-container

An alternative way to run the license-checker is using the docker container leanix/pivio-client, which has already embedded the license-checker and all required node modules. This way has the advantage that the license-checker owned dependencies will not pollute your npm-project which has to be analyzed.
Enter your npm-project and run:

docker run --rm -v $(pwd):/source leanix/pivio-client npm-license-checker

2.2 Maven

Similar to NPM, you only have to execute the following command in your project:

mvn org.codehaus.mojo:license-maven-plugin:download-licenses

Inside your target folder this generates a new folder generated-resources which contains a licenses.xml and another folder licenses which actually contains the license texts for all licenses to your dependencies.

Having completed this and put a pivio.yaml as described above in place, you can run the Pivio client and have the information transferred to LeanIX automatically.

3. Black and White Listing

You can provide a black or white list in order to filter the dependencies shown, e.g. in a JavaScript type application, you might not be interested in all the Angular versions (or only in those - who knows?). Then you can provide the Pivio client with said white or black list. If you provide both, then the white list is applied first and what is left then is filtered against the black list.

The black and white list parameters specified in the Pivio file won't be sent to the LeanIX servers.

id: leanix-pathfinder-web
api_token <api-token from leanix administration page>
name: "[autogenerated]"
type: service
owner: LeanIX front-end development
description: "[autogenerated]"
vcsroot: <your github link>
lifecycle: production
tags:
  - Angular
  - JavaScript
  - front-end
  - TypeScript
  - RE:Europe
links:
  buildchain: https://ci.leanix.net/view/Frontend/job/leanix-pathfinder-web-develop/
version: "[autogenerated]"
blacklist: 
  - ".*angular.*"

This would exclude all dependencies that have angular anywhere in their name. You can use any regular expression for filtering.

In a similar fashion it is possible to introduce a whitelist parameter to your Pivio YAML file.

4. Fields

Fields in the Pivio file are mapped to fields in the LeanIX Fact Sheet.

4.1 Field Mappings

Overview of transferred content:

Pivio Field
Application's FactSheet Field

id

externalId

name

name

description

description

version

release

owner

subscriptions on the FactSheet

tags

tags on the FactSheet

vcsroot

document link on the FactSheet

links

document link on the FactSheet

4.2 Tags

The tags field specify one or more tags which are assigned to the corresponding imported Fact Sheet in LeanIX. The structure of tags looks like:

...
tags:
- "myTag" #tag without a tag group specifier
- "myTagGroup:myTag" #tag with an optional tag group specifier
...

Resolving of tags in LeanIX

When the import runs, the system tries to find tags and its optionally tag groups in Leanix using these strategies:

  1. If no tag group specifier is used, the system tries to find the tag in the 'Other tags' tag group. If no tag is found there it reads all remaining tag groups in alphabetic order and uses the first matching tag.
  2. In case a tag group specifier is used, first a tag group is searched, using the short name or the name of the tag group to match. Only the tags within the found tag group are used to find the requested tag itself.

Tags that are not available in LeanIX

If no existing tag is found in LeanIX, a new tag will be created and assigned to the Fact Sheet.

  1. If no tag group specifier is used, the missing tag is created in 'Other tags' tag group.
  2. In case a tag group specifier is used, first the tag group is searched as described in the paragraph above. When a tag group is found, the missing tag is added into this tag group. Otherwise, a new tag group with given tag group specifier as name and short name is created and in this tag group the missing tag is added.

Searching of tags and tag groups will be done in a case insensitive way.

4.3 Subscriptions

A subscription with the type Responsible gets automatically created for the email address at the owner field. If that email address is not a assigned to a user in the workspace, the integration will generate a warning in the LeanIX synchronisation log.

4.4 Documents

All entries in the links section in Pivio are represented as documents at the Fact Sheet. Additionally the vcsroot field is added as a document Version Control to the Fact Sheet.