Field and Value Processing

Data Processors allow to map the incoming data found in the LDIF to fields of the specific entity in LeanIX (e.g. a factsheet) or the other way round in case of "Outbound Data Processors". The configuration allows to map incomiƒng values to different types of LeanIX fields (single value, float, multi value, life cycle...).

Applying Field and Value mappings may result in errors in case of source fields are not existing. Reason may be low input data quality, optional data in the source system. This types of "error" is expected and would only be noted in "test mode". Processing for next configuration lines of the data processor and next data processor always continues.

Type conversions to proper output type (variable for LeanIX, String for LDIF) happen transparently. JUEL provides implicit type conversion and allows even calculations on Strings that contain numbers.

JUEL and RegEx

To provide high flexibility, predictability and at the same time guarantee to easily understand the configuration, all relevant configuration options for the Data Processors always support a combination of JUEL (http://juel.sourceforge.net/) and regEx configuration executed after each other.

JUEL allows to access and combine all input fields and values of the incoming data and data of the target entity (e.g. fact sheet).

RegEx allows final string mapping on the JUEL result.

All conversion of data types happens transparently.

While a most simple JUEL is always required to define the value to be used as an output, the RegEx replace may be empty if no value conversion is supposed to happen. Both methods provide in parts overlapping functionality. This is wanted and allows the user to focus on a potential solution based on technical knowledge.

Fields and value mapping in the Data Processors are configured as a list of single field configurations. Each configuration allows a JUEL/RegEx for the Key and a list of JUEL/RegEx Match/RegEx Replace for the value(s). This allows multi value field support.

Each JUEL Expression returns data.

Processing of each field configuration works following this specification:

This logic allows for configurations that fit many types of scenarios.

Usage of RegEx and JUEL expressions

All RegEx filters allow negation and case insensitivity. The Java RegEx syntax can be applied: To match all but "notMe", "^((?!notMe).)*$" would be used. To ensure matching in a case insensitive manner, you'd add "(?i)" to the beginning of the regular expression.

Each inbound Data Processor JUEL expression contain the following references to the data object the is in scope for processing:

Each of them allow to access all data elements in the same or in subsections. It allows to e.g. access the id of the connector creating the LDIF. "${header.connectorId}" would result in an evaluated string "Kub Dev-001".

Using the "header" section, there is as well access to the global custom data section. Using "header.customFields.myGlodaldata1" the value of "myGlobaldata1" would be useable in any expression, given such a global value is provided in the JUEL. If not present (no customFields section or no defined key), this will always evaluate to an empty string.

Other examples with resulting string:

Users can use any type of operation that can be executed on String objects in Java. Documentation of all the Java String methods is not in scope of this documentation. Methods for Java 8 can be found here: https://docs.oracle.com/javase/8/docs/api/java/lang/String.html

Best Name for Fact Sheet use case

There are situations where it is not easy to find the best potential name for a fact sheet based on incoming data. The best name may not be available as it is not unique. Another use case might be that the source may provide different candidates for a name where we want to select from best possible option to lower ranked options automatically based on information availability for each data object.
On the other hand we want to ensure that we do not change names of already created fact sheets all of a sudden just because a better name option became available during an update.
All the above use cases can be covered simply by providing a list of potential name candidates. Every candidate that results in a null (evaluated in the 'values' section) or is already taken by another fact sheet will be skipped. In case I want to keep a name once set and not changed after creation, admins configure to read current fact sheet content and use the existing name as a first option. This will automatically be skipped if the fact sheet is not yet existing.
Please see the example:

{
  "processors": [
    {
      "processorType": "inboundFactSheet",
      "processorName": "Apps from Deployments",
      "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
      "type": "Application",
      "filter": {
        "exactType": "Deployment"
      },
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 0,
      "updates": [
        {
          "key": {
            "expr": "name"
          },
          "values": [
            {
              "expr": "${lx.factsheet.name}"
            },
            {
              "expr": "${data.app}"
            },
            {
              "expr": "${data.app2}"
            },
            {
              "expr": "${data.app3}"
            }
          ]
        }
      ],
      "read": {
        "fields": [
          "name"
        ]
      },
      "logLevel": "debug"
    }
  ]
}

You may test the above example with this demo data and play around with matching against existing Fact Sheet names and remove/rename some of the keys from the source data and do test runs:

{
  "connectorType": "ee",
  "connectorId": "Kub Dev-001",
  "connectorVersion": "1.2.0",
  "lxVersion": "1.0.0",
  "description": "Imports kubernetes data into LeanIX",
  "processingDirection": "inbound",
  "processingMode": "partial",
  "customFields": {},
  "content": [
    {
      "type": "Deployment",
      "id": "634c16bf-198c-1129-9d08-92630b573fbf",
      "data": {
        "app3": "veryLongAndUnhandyNameIDoNotWantToSeeIfPossible",
        "app2": "littleBitBetterNameButStillNotGood",
        "app": "Best Name"
      }
    }
  ]
}

Examples how to use JUEL in a more advanced fashion:

Working with Expressions using "integration"

To test the below example, please change the id in "ids" to an existing internal id of a fact sheet in your workspace. You may just open a fact sheet and copy the id from the browser URL.
In real world scenarios, you may not want to export the whole object, but iterate over tag groups or export a subset of the information.

{
	"scope": {
		"ids": [
			"869ee28b-c60a-4e88-8d18-f9e4ff466456"
		],
		"facetFilters": []
	},
	"processors": [
		{
			"processorType": "outboundFactSheet",
			"processorName": "Export tag groups and tags",
			"processorDescription": "Sample how to export all available tag groups and all tags as part of one fact sheet export",
			"fields": [
				"name"
			],
			"output": [
				{
					"key": {
						"expr": "content.id"
					},
					"values": [
						{
							"expr": "${lx.factsheet.id}"
						}
					]
				},
				{
					"key": {
						"expr": "content.type"
					},
					"values": [
						{
							"expr": "${lx.factsheet.type}"
						}
					]
				},
				{
					"key": {
						"expr": "name"
					},
					"values": [
						{
							"expr": "${lx.factsheet.name}"
						}
					]
				},
				{
					"key": {
						"expr": "tagGroupsAndTags"
					},
					"values": [
						{
							"object": "${integration.tags.getAllTagGroups()}"
						}
					]
				}
			]
		}
	]
}

A similar configuration will help gathering all tag groups where a specific fact sheet has tags set. The lx.tagGroups list will be filled with all tag groups where the Fact Sheet has at least one tag set. And inside each tag group element there will be a list of the found tags for the Fact Sheet. The below example filters and returns only a sub set (default tag group). Just adding "${true}" in the filter will ensure to return all tag groups and included tags the Fact Sheet has set:

{
	"scope": {
		"ids": [
			"bb8b0b74-f737-4f1b-a937-a06bddf3fe47"
		],
		"facetFilters": [
			{
				"keys": [
					"Application"
				],
				"facetKey": "FactSheetTypes",
				"operator": "OR"
			}
		]
	},
	"processors": [
		{
			"processorType": "outboundFactSheet",
			"processorName": "Export to LDIF",
			"processorDescription": "This is an example how to use the processor",
			"enabled": true,
			"fields": [
				"lifecycle",
				"location",
				"createdAt",
				"technicalSuitabilityDescription",
				"description"
			],
			"relations": {
				"filter": [
					"relToParent",
					"relApplicationToITComponent"
				],
				"fields": [
					"description"
				],
				"targetFields": [
					"displayName",
					"externalId"
				],
				"constrainingRelations": false
			},
			"tags": {
				"groups": [
					"SomeTagGroupName"
				],
				"multipleGroups": "${dm.tagGroup.name =='Other tags'}"
			},
			"subscriptions": {
				"types": [
					"RESPONSIBLE"
				]
			},
			"documents": {
				"filter": ".*"
			},
			"output": [
				{
					"key": {
						"expr": "content.id"
					},
					"values": [
						{
							"expr": "${lx.factsheet.id}"
						}
					]
				},
				{
					"key": {
						"expr": "content.type"
					},
					"values": [
						{
							"expr": "${lx.factsheet.type}"
						}
					]
				},
				{
					"key": {
						"expr": "Description"
					},
					"values": [
						{
							"expr": "${integration.toJson(lx.tagGroups).toString()}"
						}
					],
					"optional": true
				}
			]
		}
	]
}

Calculating with single-select fields

{
    "connectorId": "id-92476445-10b3-40f7-9386-6f13c61e4b89",
    "connectorType": "ee",
    "connectorVersion": "1.2.0",
    "processingDirection": "inbound",
    "processingMode": "partial",
    "processors": [
        {
            "enabled": true,
            "filter": {
                "type": "DataObject"
            },
            "identifier": {
                "internal": "${content.id}"
            },
            "logLevel": "debug",
            "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
            "processorName": "Apps from Deployments",
            "processorType": "inboundFactSheet",
            "read": {
                "fields": [
                    "businessValue",
                    "projectRisk",
                    "dataClassification"
                ],
                "noNullForOrdinal": true
            },
            "run": 0,
            "type": "DataObject",
            "updates": [
                {
                    "key": {
                        "expr": "description"
                    },
                    "values": [
                        {
                            "expr": "No null - ${lx.toOrdinal('dataClassification')}"
                        }
                    ]
                }
            ],
            "variables": [
                {
                    "key": "deploymentMaturity",
                    "value": "${data.maturity}"
                }
            ]
        }
    ]
}

Removing content from fields using the Integration API

The integration API can be used to remove content. In case Values array contains "null" values after evaluating all configured elements in the Values Array, the Integration API will try to reset the configured field to an initial "not filled" state. This is specifically helpful for Single or multi select fields. String fields can simply be cleaned by passing an empty String. Numbers may rather be set to 0.
To avoid a warning, that no value could be found, ensure the "optional" field is used.

📘

variableProcessor

Is used to only write values to internal variables. This will be used for aggregation use cases where the LDIF content needs to be used to only collect values without directly writing anything to LeanIX.

Using current Fact Sheet information when writing to a Fact Sheet

All inbound processors allow to read information from the fact sheet (currently supported: fields, relations, subscriptions, tags, documents and metrics) and use the information when writing back to the fact sheet. The below example shows two use case examples, where a cost field is increased by the incoming value and an update of the risk section will only be done if the description is not starting with a key word "manually".
The example as well contains information how to use this feature.

{
  "processors": [
    {
      "processorType": "inboundFactSheet",
      "processorName": "Apps from Deployments",
      "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
      "type": "Project",
      "filter": {
        "exactType": "prj"
      },
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 0,
      "updates": [
        {
          "key": {
            "expr": "budgetOpEx"
          },
          "values": [
            {
              "expr": "${lx.factsheet.budgetOpEx+data.monthlyOpEx}"
            }
          ]
        },
        {
          "key": {
            "expr": "projectRisk"
          },
          "values": [
            {
              "expr": "${(lx.tags.toString().contains('\"name\":\"MANUAL_INPUT'))?null:data.risk}",
              "regexMatch": ".+"
            }
          ],
          "optional": true
        },
        {
          "key": {
            "expr": "projectRiskDescription"
          },
          "values": [
            {
              "expr": "${(lx.tags.toString().contains('\"name\":\"MANUAL_INPUT'))?null:data.riskDescription}",
              "regexMatch": ".+"
            }
          ],
          "optional": true
        },
        {
          "key": {
            "expr": "metrics"
          },
          "values": [
            {
              "expr": "${integration.toJson(lx.metrics.variableName.values)}"
            }
          ]
        }      
      ],
      "logLevel": "debug",
      "read": {
        "fields": [
          "budgetOpEx"
        ],
        "tags": {
          "groups": [
            "Other tags"
          ]
        },
        "relations": {
          "filter": [
            "relToParent",
            "relApplicationToITComponent"
          ],
          "fields": [
            "description"
          ],
          "targetFields": [
            "displayName",
            "externalId",
            "location"
          ]
        },
        "subscriptions": {
          "types": [
            "RESPONSIBLE"
          ]
        },
        "metrics": [
            {
                "name": "variableName",
                "measurement": "money",
                "fieldName": "dollars_per_day",
                "aggregationFunction": "MEAN",
                "groupBy": "1h",
                "start": "2020-01-20T00:00:00Z",
                "duration": "P0DT24H30M",
                "rules": {
                    "key": "factSheetId",
                    "comparator": "=",
                    "compareWith": "${lx.factsheet.id}"
                }
            }
        ],
        "impacts": {
      		"readAll": true
				}
       }
    }
  ]
}

Example LDIF to test the above processor. The workspace needs to contain a Project Fact Sheet with external ID "12345". Or change the LDIF data to an external ID of a Project Fact Sheet existing in the workspace:

{
  "connectorType": "showcaseUpdate",
  "connectorId": "showcaseUpdate",
  "connectorVersion": "1.0.0",
  "lxVersion": "1.0.0",
  "content": [
    {
      "type": "prj",
      "id": "12345",
      "data": {
        "monthlyOpEx": 50000,
        "risk": "lowProjectRisk",
        "riskDescription": "The risk is considered to be low."
      }
    }
  ]
}

📘

Access to fields on relations and relation target fields

By defining the fields on the relations and on the target fact sheets of a relation, admins can use the values in JUEL expressions in the output section. The found relations need to be iterated using "forEach". Each element will then contain the standard information about e.g. name and type of a relation plus the requested fields. They can be accesses following the structure shown in the below example

"lx.relationsElement": {
						"id": "9316291b-361a-4050-ac79-bf9f96811fb1",
						"type": "relApplicationToITComponent",
						"target": {
							"id": "161abc0d-7bed-4440-b756-5c14a741e1ad",
							"name": "Application Development",
							"type": "ITComponent"
						},
						"activeFrom": "2021-01-18",
						"description": ""
					}

Dynamic definition of Fact Sheet fields to read data from

Sometimes it is helpful to decide at run time which fields from a Fact Sheet to read and not hard code the names of the fields in the configuration.
For this purpose, Integration API allows to define a key "multipleFields" and a value that is a JUEL expression and will be resolved to boolean true and false. The Integration API will iterate over all available fields taken from the data model and allow the expression to do any filter logic required. As input value, the currently iterated field can be used with "dm.factSheetField.name". The type of the field can be identified with "dm.factSheetField.type"
In following JUEL expressions like forEach or update section, the list of read fields can be used with dm.factSheetFields, which is an object with keys: name, type and factsheetType
The below example would read all fields of type "STRING" from a fact sheet.

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Apps from Deployments",
			"processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
			"type": "Application",
			"filter": {
				"exactType": "Deployment"
			},
			"identifier": {
				"external": {
					"id": {
						"expr": "${content.id}"
					},
					"type": {
						"expr": "externalId"
					}
				}
			},
      "read": {
				"multipleFields": "${dm.factSheetField.type=='STRING'}"
			}
			"updates": [
				{
					"key": {
						"expr": "${integration.valueOfForEach.name}"
					},
					"values": [
						{
							"expr": "${lx[integration.valueOfForEach.name]} - added by Integration API"
						}
					]
				}
			],
			"forEach": "${dm.factSheetFields}",
			"logLevel": "debug"
		}
	]
}

In the sample, all String fields of the fact sheet get a string " - added by Integration API" appended.
multipleFields can be used for relations as well.
After evaluation of dynamic relation reading, fields dm.relationFilters[], lx.relationTargetFields[] and lx.relationFields[] are available

"multipleFields": "${data.fieldsToRead.contains(dm.relationField.name) && dm.relationField.type=='relToParent'}"

          "multipleTargetFields": "${data.fieldsToRead.contains(dm.factSheetField.name) && dm.relationType=='relToParent'}"

          "multipleFilters": "${data.myDynamicRelationList.contains(dm.relationType)}",

Same way, dynamic definition of Tags can happen: (after execution, the collected tag groups are available in a "lx.tagGroups" list.
In the filter "multipleGroups", the object of the currently iterated tag group "dm.tagGroup" can be used

{
	"read": {
		"tags": {
			"multipleGroups": "${true}"
			"groups": [
				"Cloud Transformation"
			]
		}
	}
}

🚧

Availability of information read from the Fact Sheet

Information read from the fact sheet is available in the output section. The information is not available in the outer forEach, in the identifier and the filter section. The reason for this is, that at the time when the content in these sections is evaluated, the target Fact Sheet is not yet identified.

Automatic deletion when loading data using inbound Data Processors

Integration API supports the processing mode "full" mode when creating the configuration. Only in case, the configuration is set to mode "full", a section "deletionScope" is read from the processor configuration. The following operations are supported:

• Deletion of Fact Sheets If that section does contain a key "factSheets", all fact sheets matching the scope query inside will be removed if they are not found in the processed LDIF.
  All Fact Sheets that match the deletion scope but are not touched by an inbound Data Processor during processing, will be removed (set to "Archived")
• Relations can be automatically removed as well. The structure to define relations to be deleted is similar. See an example configuration below. The example removes all relations but by narrowing the scope to fewer fact sheets, only for these fact sheets relations will be removed.
• Documents can be deleted by defining a scope of fact sheets and adding a regular expression pattern to match the documents by name that may be removed if they are no longer referenced by the incoming LDIF data.

How does this work? Do I need to delete any data manually or first delete in a processor?

No.
The concept of deletion is different in Integration API. There is no active deletion needed by the user. All the deletion is done by the iAPI automatically and transparently.
In an inbound syncRun you just sent fact sheets an other data like tags or relations to create or update them. Both, creation and updating, will cause the fact sheets and other data to be “touched” by this whole syncRun.
Before finishing the whole syncRun (after last run level execution), iAPI deletes all factsheets that are defined to be in the “deletion scope” but have not been touched at all.
This avoids e.g. constant “deleted-created-deleted-creatred” cycles that would be visible in the audit log of fact sheets and avoids any manual work deletion any data.

📘

Sending data for an archived FactSheets in the SyncRun

In case an archived FactSheet exists in the Workspace with the same externalId as sent in one of the input LDIFs, Integration API will create a new FactSheet rather than recovering the deleted FactSheet, this is done in order to avoid unexpected data on the archived FactSheet reflected on the Active FactSheet.
PS: This scenario can only happen when the "uniqueFactSheet" attribute for a FactSheet is set to "false" when set to "true" externalIds are already cleared while archiving a FactSheet.

📘

Multiple deletion scopes

Please note that you can define multiple sets of deletion scopes for every type (e.g. 2 fact sheet deletion scopes and 3 relation deletion scopes). Processed items during synchronization runs will be compared against each set separately. Any item in each deletion scope definition will be removed if not touched during processing. It is allowed to even define overlapping scopes. Each item will be handled once only.

Example Configuration that removes all Projects from the LeanIX EA workspace that are no longer part of the incoming LDIF data, all relations from Applications to ITComponents and all documents with prefix "MyDocs_".

{
  "deletionScope": {
		"maximumDeletionRatio": {
				"relations": 40,
				"factSheets": 30
		},
    "factSheets": [
      {
        "scope" : {
          "facetFilters": [
            {
              "keys": [
                "Project"
              ],
              "facetKey": "FactSheetTypes",
              "operator": "OR"
            }
          ],
          "ids": []
        }
      }
    ],
    "relations": [
      {
        "relationTypes": [
          "relApplicationToITComponent"
        ],
        "scope": {
          "facetFilters": [],
          "ids": []
        }
      }
    ],
    "documents": [
      {
        "documentMatches": [
          "^MyDocs_.*"
        ],
        "scope": {
          "facetFilters": [
            {
              "keys": [
                "Project"
              ],
              "facetKey": "FactSheetTypes",
              "operator": "OR"
            }
          ],
          "ids": []
        }
      }
    ]
  },
  "processors": [
    {
      "processorType": "inboundFactSheet",
      "processorName": "Apps from Deployments",
      "processorDescription": "Creates LeanIX Applications from Deployments",
      "type": "Project",
      "filter": {
        "exactType": "prj"
      },
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 0,
      "updates": [
        {
          "key": {
            "expr": "name"
          },
          "values": [
            {
              "expr": "${data.name}"
            }
          ]
        }
      ],
      "enabled": true,
      "logLevel": "debug"
    }
  ]
}

LDIF to try out deletion:

{
  "connectorType": "prjFull",
  "connectorId": "prjFull",
  "connectorVersion": "1.0.0",
  "lxVersion": "1.0.0",
  "content": [
    {
      "type": "Project",
      "id": "prj-42",
      "data": {
        "name": "Project 42"
      }
    },
    {
      "type": "Project",
      "id": "prj-43",
      "data": {
        "name": "Project 43"
      }
    },
    {
      "type": "Project",
      "id": "prj-44",
      "data": {
        "name": "Project 44"
      }
    }
  ]
}

To try out, please execute multiple times to first create all projects, then remove one item and try again.

❗️

Example will delete all projects

Executing this example needs to be done with care. All potentially existing project Fact Sheets in the workspace will be in scope for deletion. To limit, you may want to change the sample that a tag "TEST_PRJ" or similar will be set for the test projects. This tag can be added as filter criteria to the deletion scope definition

Example configuration to remove tags from fact sheets

Deletion of Tags works similar. One or more deletion scope sections of type "tags" needs to be configured. The scope defines the set of fact sheets to be looked at when removing tags and allows to configure a tag group and a tag name to be deleted. Tag names support regular expression matching to allow removal of tags based on name patterns. Please note that tags will be removed from the fact sheets where no longer referenced by processors adding them but tags themselves will not be deleted.

{
  "deletionScope": {
		"maximumDeletionRatio": {
				"tags": 40
		},
    "tags": [
      {
        "tagScopes": [
          {
            "group": "myGroup",
            "tag": "Prefix_.*"
          }
        ],
        "scope": {
          "facetFilters": [
            {
              "keys": [
                "Project"
              ],
              "facetKey": "FactSheetTypes",
              "operator": "OR"
            }
          ],
          "ids": []
        },
        "advanced": "${lx.tag.tagGroup.name==null}"
      }
    ],
		"subscriptions": [
			{
				"subscriptionScopes": [
					{
						"type":"RESPONSIBLE",
						"roles": [
							"My Role"
						]
					}
				],
				"scope": {
					"facetFilters": [
						{
							"keys": [
								"Project"
							],
							"facetKey": "FactSheetTypes",
							"operator": "OR"
						}
					],
					"ids": [

					]
				}
			}
		]
  },
  "processors": []
}

Example configuration to use advanced deletion on subscriptions. Here to remove all subscriptions on application fact sheets that have an anonymised user as a subscriber for a subscription of type "RESPONSIBLE":

{
    "deletionScope": {
        "subscriptions": [
            {
                "subscriptionScopes": [
                    {
                        "type": "RESPONSIBLE",
                        "roles": []
                    }
                ],
                "scope": {
                    "facetFilters": [
                        {
                            "keys": [
                                "Application"
                            ],
                            "facetKey": "FactSheetTypes",
                            "operator": "OR"
                        }
                    ]
                },
                "advanced": "${lx.subscription.user.userName=='AnonymizedUser'}"
            }
        ]
    },
    "processors": []
}

Please note, that the above configuration does not configure any processors, thus can work with an empty LDIF as input to trigger Integration API:

{
  "connectorId": "subscription",
  "connectorType": "subscription",
  "connectorVersion": "1.0.0",
  "content": [],
  "lxVersion": "1.0.0"
}

Protection against removal of items

The Integration API can be configured to not delete any configured item if the ratio of to be deleted items in any deletion scope exceeds a defined ratio compared to the total items in scope for the deletion scope. This can be used to protect the existing data in case of erroneously sending incomplete data into a full scope synchronisation with deletion defined. by defining "maximumDeletionRatio" for each type of deletion scope, this mechanism can be used. A format example can be found in the examples above. A threshold of 30 means that all deletion will be stopped if 30% or more items would be deleted for a given deletion scope.

📘

Valid content for the deletion scope

To create valid JSON content to define the scope of Fact Sheets to be deleted if they no longer exist in the incoming LDIF, admins may want to use an outbound configuration. Using this configuration, a Button "Scope" is available that opens the facet filter UI. Once confirmed, the scope is automatically pasted to the processor configuration. Admins may copy and paste it into the inbound configuration where they need to use automatic deletion.

Advanced deletion options

The functionality to delete content for elements we do no longer see references in the LDIF, can be used in an even more advanced was if required. The deletion scope for Relations, Tags and Documents may contain an optional key "advanced". If configured in the specific deletion scope, the value is evaluated as a JUEL expression resulting in "true" or "false". Only such elements will be added to the scope of elements potentially to be deleted if not touched where the JUEL expression evaluates to "true".
Multiple deletion scopes of same or different types can be defined. Please note that deletions of Fact Sheets will always happen last. This allows to use the Fact Sheet meta data lx.factsheet.* even for Fact Sheets that will be deleted. If allows to even use the owner field of the fact sheet deletion and evaluate in a relation, tag, document or subscription deletion scope in the state before the owner of the current run will be removed from the field (see ownership concept of advanced Fact Sheet deletion)
Please note, that currently no content from fields of type projectStatus is available for advanced deletion.

The information about the current Fact Sheet (for relations always the source Fact Sheet of a relation) is available using "lx.factsheet.*". All fields of the Fact Sheet can be used. In addition, all meta data fields of relations, documents and tags can be used in the JUEL for the related type of deletion scope.

{
	"deletionScope": {
		"factSheets": [
			{
        "scope": [
				  "facetFilters": [
					  {
						  "keys": [
							  "Project"
		  				],
			  			"facetKey": "FactSheetTypes",
				  		"operator": "OR"
					  }
				  ]
        ]
			}
		],
		"relations": [
			{
				"relationTypes": [
					"relProjectToITComponent"
				],
				"scope": {
					"ids": [],
					"facetFilters": []
				},
        "advanced": "${lx.relation.description.contains('from hr service')}"
			}
		],
		"documents": [
			{
				"documentMatches": [
					".*"
				],
				"scope": {
					"facetFilters": [
						{
							"keys": [
								"Project"
							],
							"facetKey": "FactSheetTypes",
							"operator": "OR"
						}
					]
				},
				"advanced": "${lx.document.documentType.equals('jira') || lx.document.name.equals('someName') }"
			}
		]
	},
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Read Projects",
			"processorDescription": "Creates LeanIX Projects from Project Management Solution",
			"type": "Project",
			"filter": {
				"exactType": "prj"
			},
			"identifier": {
				"external": {
					"id": {
						"expr": "${content.id}"
					},
					"type": {
						"expr": "externalId"
					}
				}
			}
		}
	]
}

Deletion option for multiple external sources

Advanced deletion is as well available for Fact Sheets but works slightly different. Advanced Fact Sheet deletion supports multiple external sources for one LeanIX Fact Sheet. Deletion would not happen unless the last referenced source of a fact sheet does not longer contain information about the fact sheet.
This functionality is helpful in cases where a LeanIX Fact Sheet might be created and again removed by potentially more than one foreign system providing separate LDIF to update the LeanIX side.
In cases, where the Fact Sheet was no longer referenced by one of the sources, a deletion would not be a valid solution unless all sources no longer contain the information.
For such cases, the advanced deletion allows every sources to set a unique id as a marker. If this marker, called "owner", was found in the configuration, the Integration API first checks the field with all markers and only removes (archives) the Face Sheet if the list of markers found in the field is empty. The field to store the markers needs to be created in the data model as a standard String type. Integration API will read the content ant treat as a JSON list.
The below example shows example usage including the way to add the marker for a specific owner as part of the output section.
Please note that the example uses the "alias" field to store the owner information. This is for testing and demonstration as it allows easy inspection and required no data model work. For production usage, this needs to be written to a new created field not visible in the UI.

{
	"deletionScope": {
		"factSheets": [
			{
				"scope": {
					"ids": [],
					"facetFilters": [
						{
							"keys": [
								"Process"
							],
							"facetKey": "FactSheetTypes",
							"operator": "OR"
						}
					]
				},
				"owner": {
					"fieldName": "alias",
					"ownerId": "myOwner"
				}
			}
		]
	},
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Apps from Deployments",
			"processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
			"type": "Process",
			"identifier": {
				"external": {
					"id": {
						"expr": "fullSyncOwnerTest ${integration.valueOfForEach}"
					},
					"type": {
						"expr": "externalId"
					}
				}
			},
			"run": 0,
			"updates": [
				{
					"key": {
						"expr": "name"
					},
					"values": [
						{
							"expr": "Full Sync Owner Test ${integration.valueOfForEach}"
						}
					]
				},
				{
					"key": {
						"expr": "alias"
					},
					"values": [
						{
							"expr": "${helper:addIfNotExisting(lx.factsheet.alias, 'myOwner')}"
						}
					]
				}
			],
			"enabled": false,
			"forEach": "${data.tags}",
			"logLevel": "debug",
			"read": {
				"fields": [
					"alias"
				]
			}
		}
	]
}

Use case: Remove all tags of two tag groups from Fact Sheets of a given type can be performed as easy as shown in the below example. The deletion scope marks all Fact Sheets with tags in the tag groups and the given Fact Sheet type to be deleted if not touched. Then the processor configuration stays empty. No Fact Sheet will be touched when processing. At the end of the run, all tags will be removed. Please do not forget to increase the deletion limit set to 50% by default to 101 to allow removing even 100% of the tags.

{
  "deletionScope": {
		"maximumDeletionRatio": {
				"tags": 101
		},
    "tags": [
      {
        "tagScopes": [
          {
            "group": "Cloud:Region",
            "tag": ".*"
          }
        ],
        "scope": {
          "facetFilters": [
			{
				"facetKey": "FactSheetTypes",
				"operator": "OR",
				"keys": [
					"CloudComponent"
				]
			}
		],
          "ids": []
        }
      },{
        "tagScopes": [
          {
            "group": "Cloud:Cloud Service",
            "tag": ".*"
          }
        ],
        "scope": {
          "facetFilters": [
			{
				"facetKey": "FactSheetTypes",
				"operator": "OR",
				"keys": [
					"CloudComponent"
				]
			}
		],
          "ids": []
        }
      },{
        "tagScopes": [
          {
            "group": "Cloud:Tech Category",
            "tag": ".*"
          }
        ],
        "scope": {
          "facetFilters": [
			{
				"facetKey": "FactSheetTypes",
				"operator": "OR",
				"keys": [
					"CloudComponent"
				]
			}
		],
          "ids": []
        }
      }
    ]
  },
  "processors": []
}

📘

Using External IDs in pathfinder search scopes

When working with integrations, specifically with deletion scopes it is handy to know that pathfinder is capable filtering documents by their external id. and not only by internal id which is most of the times not known to foreign systems.
To filter by external ids, just use the field "externalIds" instead of "ids" in the search scope definition.
Please note that Pathfinder required a special syntax when defining external ids using the name of the externalID field, a slash ("/") then value of the externalId.
Example for the default externalId field:
"externalIds": ["${'externalId/'.concat(header.customFields.myExternalId)}"]

Exchanging Data between Data Objects and Aggregation

In some situations it may be required to use information from multiple Data Objects and store a joint result in another entity like a Fact Sheet or Relation. Even Creating specific relations if we find certain value combinations in different Data Objects is possible.

In order to perform such operations, a "variables" section is available to write and add to, while iterating over Data Objects. Data Processors in the following Runs (!! Not in the same Run) can then read the values and perform defined operations on them.

This works in the following steps:

Writing Variables using Expressions

"variables": [
  {
    "key": "prefix_§{dataMyNameFromDataObjectValue}"
      "value": "${data.myValueFromDataObject}"
  }
]

Example Processor Writing Variables

Below is an example of a processor with a matching LDIF that shows how variables work. In the first run which is marked by the processor with "run": 0, the variable section can be called on with the key aggregatedCosts and gathers together all the costs in the data section of each entry of an LDIF that is gathered by the filter in place. In this case, that filter is a fact sheet of type ITComponent. In the next run marked with "run": 1, the processor is calling the sum function on the variable aggregatedCosts and writing the sum to the description field of the fact sheets that fall under the specified filter, which in this case is all the Applications in the LDIF.

{
  "processors": [
    {
      "processorType": "inboundFactSheet",
      "processorName": "Create IT Components",
      "processorDescription": "One Processor for IT Components",
      "enabled": true,
      "type": "ITComponent",
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 0,
      "filter": {
        "exactType": "ITComponent"
      },
      "updates": [
        {
          "key": {
            "expr": "name"
          },
          "values": [
            {
              "expr": "${data.name}"
            }
          ]
        }
      ],
      "variables": [
        {
          "key": "aggregatedCosts",
          "value": "${data.cost}"
        }
      ]
    },
    {
      "processorType": "inboundFactSheet",
      "processorName": "Create Applications",
      "processorDescription": "Aggregated IT Costs in Application's Description",
      "enabled": true,
      "type": "Application",
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 1,
      "filter": {
        "exactType": "Application"
      },
      "updates": [
        {
          "key": {
            "expr": "name"
          },
          "values": [
            {
              "expr": "${data.name}"
            }
          ]
        },
        {
          "key": {
            "expr": "description"
          },
          "values": [
            {
              "expr": "${variables.aggregatedCosts.sum()}"
            }
          ]
        }
      ]
    }
  ],
  "variables": {}
}

{
  "connectorType": "ee",
  "connectorId": "Kub Dev-001",
  "connectorVersion": "1.2.0",
  "lxVersion": "1.0.0",
  "content": [
    {
      "type": "ITComponent",
      "id": "itc1",
      "data": {
        "name": "IT1",
        "cost": 5
      }
    },
    {
      "type": "ITComponent",
      "id": "itc2",
      "data": {
        "name": "IT2",
        "cost": 6
      }
    },
    {
      "type": "Application",
      "id": "app",
      "data": {
        "name": "My App"
      }
    }
  ]
}

The result is three fact sheets created, two IT Components, and one Application with a description of 11. Note that the costs of the IT Components were not written in the IT Component's fact sheets.

Dynamic Variable Handling

${variables[integration.valueOfForEach.concat('_cost')].sum()}
(which is same as: variables['12345_cost'].sum() in case valueOfForEach is "12345")

Supported Dynamic Variable Operations

Supported operations are listed below. Each invalid entry will be counted as "0" when calculating.

ForEach Logic

Each data processor provides additional capabilities to handle values that are lists. Using the standard functionality, every data processor will be executed exactly one time for each data object sent to the Integration API.

Sometimes however, there is a need to update multiple factsheets or multiple fields in a fact sheet for each value we find in a list of values found in the LDIF.

...
"data": {
  "attachedDocuments": [
    {
      "extension": "vsdx",
      "name": "thediagram.vsdx",
      "displayName": "Diagram",
      "url": "sotrage.azure.com/123/thediagram.vsdx",
      "content": null
    },
    {
      "extension": "docx",
      "name": "thedoc.docx",
      "displayName": "Documentation",
      "url": "sotrage.azure.com/123/thedoc.docx",
      "content": null
    },
    {
      "extension": "html",
      "name": "webpage.html",
      "displayName": "Web Page",
      "url": null,
      "content": "<body>the vm 789 ...</body>"
    }
  ]
  "version": "1.8.4",
  "myForEachField": "attachedDocuments",
  "maturity": "3",
  "note": "I did the first comment here",
  "Home Country": "D",
  "Other Country": "UK",
  "clusterName": "leanix-westeurope-int-aks"
}
...

{
  "processorType": "inboundFactSheet",
  "processorName": "Deployment To Application",
  "processorDescription": "The processor creates or updates an Application from every data object of type 'Deployment'",
  "type": "Application",
  "name": "My Awesome App",
  "run": 0,
  "enabled": true,
  "identifier": {
    "external": {
      "id": {
        "expr": "${content.id}"
      },
      "type": {
        "expr": "externalId"
      }
    }
  },
  "filter": {
    "exactType": "Deployment"
  },
  "forEach": "${data.myForEachField}",
  "updates": [
    {
      "key": {
        "expr": "name"
      },
      "values": [
        {
          "expr": "${data.attachedDocuments[integration.indexOfForEach].name}", // or in short: ${integration.valueOfForEach.name}
          "regexReplace": {
            "match": "",
            "replace": ""
          }
        },
        {
          "expr": "${data.value}"
        }
      ]
    }
  ]
}

Using the "forEach" section in each data processor as in the example above. Will result in executing the data processor "Deployment To Application" four times for the given data object and each run will allow the user to use the index of the current iteration in all expressions (integration.indexOfForEach).

To fill some output field of the data processor with the specific url (see example above), the configuration would look like this: ${data.attachedDocuments[integration.indexOfForEach].name}. This will generate the three different names of the attached documents in each run of the data processor. This could be used to create separate factsheets and relations from the source data.

There is another way to access the value of the element referenced by the current index:

${integration.valueOfForEach}

Which is the same as:

${data.attachedDocuments[integration.indexOfForEach].name}

The index variable however can be used to reference the same index of another list element e.g. Important note: The admin can configure a "regexReplace" section in the forEach section. This will allow to manipulate the JSON representation of the value object resulting from the expression. In case such a manipulation is configured, it will have impact on the "integration.valueOfForEach" and not alter the original data one may reference using the indexOfForEach variable in the original data and reference manually.

Of course, the logic could be used to always execute a data processor n times. Just add '[1,2,3]' as configuration and the data processor will execute three times with the index variable integration.indexOfForEach set to 0-2 for reference.

In case the field 'attachedDocuments' is not available or contains an empty list, the data processor will not execute (operate on an empty list). In case the url is a single value and no list, the data processor will execute once.

The integration API allows to iterate over list values and map values. In case of iterating over a map, indexOfForEach will always return -1 as maps are not sorted. For maps there is an additional variable "keyOfForEach" available providing access to the name of the key. The value will be accessed with "valueOfForEach"

{
  "connectorType": "ee",
  "connectorId": "Kub Dev-001",
  "connectorVersion": "1.2.0",
  "lxVersion": "1.0.0",
  "content": [
    {
      "type": "Deployment",
      "id": "634c16bf-198c-1129-9d08-92630b573fbf",
      "data": {
        "app": "HR Service",
        "version": "1.8.4",
        "myList": [
          "lValue1",
          "lValue2"
        ],
        "myMap": {
          "key1": "value1",
          "key2": "value2"
        }
      }
    }
  ]
}

{
  "processors": [
    {
      "processorType": "inboundFactSheet",
      "processorName": "Apps from Deployments",
      "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
      "type": "Application",
      "filter": {
        "exactType": "Deployment"
      },
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 0,
      "updates": [
        {
          "key": {
            "expr": "name"
          },
          "values": [
            {
              "expr": "${data.app}"
            }
          ]
        },
        {
          "key": {
            "expr": "description"
          },
          "values": [
            {
              "expr": "${integration.keyOfForEach}: ${integration.valueOfForEach}"
            }
          ]
        }
      ],
      "forEach": "${data.myMap}",
      "logLevel": "debug"
    }
  ]
}

For each logic can as well be applied inside the value section for and update key. The list or map will be iterated and the values result will contain n entires that are then mapped to the defined key. Please ensure to set the mode to "list" if not only the first value is to be used as a result (default mode for every key in the update section of a processor is "selectFirst" to only take the first non null result from what was defined in the values array. The "inner forEach" is behaving exactly as if the admin defined a fix number of elements in the "values" section.

{
					"key": {
						"expr": "targetITComponents"
					},
					"mode": "list",
					"values": [
						{
							"forEach": {
								"elementOf": "${lx.relations}",
								"filter": "${true}"
							},
							"map": [
								{
									"key": "id",
									"value": "${integration.output.valueOfForEach.target.id}"
								},
								{
									"key": "type",
									"value": "${integration.output.valueOfForEach.target.type}"
								},
								{
									"key": "name",
									"value": "${integration.output.valueOfForEach.target.displayName}"
								}
							]
						}
					]
				}

Given the processor was configured to read relations (read section) and put the results into a list "lx-relations", the above example of an outboundFactsheet processor, will output all relation results into an array as value of a key named "targetITCmponents".
Please note that admins may configure the filter JUEL expression evaluating to boolean in order to not have some of the input list elements in the output. The JUEL may contain references to "integration.output.valueOfForEach" and filter on any content.

A third option to iterate using "forEach" is to add the key at the level of "updates". It allows to create a dynamic set of field updates to be pushed to e.g. a fact sheet. In the below example the fields to be updated will be read from the incoming LDIF. In order to execute the example, the referenced Fact Sheet needs to already exist. Most easy way is to one time execute the "starter example" configuration on the workspace.

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Apps from Deployments",
			"processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
			"type": "Application",
			"filter": {
				"exactType": "Deployment"
			},
            "logLevel": "debug",
			"identifier": {
				"external": {
					"id": {
						"expr": "${content.id}"
					},
					"type": {
						"expr": "externalId"
					}
				}
			},
			"updates": [
				{
					"key": {
						"expr": "${integration.updates.keyOfForEach}"
					},
					"values": [
						{
							"expr": "${integration.updates.valueOfForEach}"
						}
					],
					"forEach": {
						"elementOf": "${data}",
						"filter": "${integration.updates.valueOfForEach!='toBeFiltered'}"
					}
				}
			]
		}
	]
}

{
	"connectorType": "Kubernetes",
	"connectorId": "Kub Dev-001",
	"connectorVersion": "1.2.0",
	"lxVersion": "1.0.0",
	"description": "Imports Kubernetes data into LeanIX",
	"processingDirection": "inbound",
	"processingMode": "partial",
	"customFields": {},
	"content": [
		{
			"type": "Deployment",
			"id": "634c16bf-198c-1129-9d08-92630b573fbf",
			"data": {
				"name": "HR Service",
				"version": "toBeFiltered",
				"description": "test description"
			}
		}
	]
}

Using all three options to iterate with "forEach" functionality, Integration API now allows to ingest data in LDIF where data structures are nested up to three levels.

Use "object" key to output all objects of eventually unknown type

The "object" key might be used instead of "map" or "expr" to retrieve a representation of any potential input object as a defined value. This allows to easily export all information without need to know about the details inside the object.

{
  "scope": {
    "facetFilters": [
      {
        "facetKey": "FactSheetTypes",
        "operator": "OR",
        "keys": [
          "Application"
        ]
      }
    ],
    "ids": ["90a8296c-92fe-4009-a4cf-21db710719ec"]
  },
  "processors": [
    {
      "processorType": "outboundFactSheet",
      "logLevel": "debug",
      "fields": ["lifecycle"],
      "output": [
        {
          "key": {
            "expr": "content.id"
          },
          "values": [
            {
              "expr": "${lx.factsheet.id}"
            }
          ]
        },
        {
          "key": {
            "expr": "content.type"
          },
          "values": [
            {
              "expr": "${lx.factsheet.type}"
            }
          ]
        },
        {
          "key": {
            "expr": "description"
          },
          "values": [
            {
              "object": "${lx.factsheet.lifecycle}"
            }
          ]
        }
      ]
    }
  ]
}

Filter Processor execution based on current Fact Sheet content

Using the onRead filter in an inbound processor allows to execute a processor based on currently existing Fact Sheets. The processor can be configured to only execute if a Fact sheet already exists or if the Fact sheet has defined values in some fields. Or exactly the other way round: The Processor may only be executed if the Fact Sheet does not yet exist. The following Example shows a Processor that will execute if a Fact Sheet exists that has a defined name and is flagged with a certain Tag.

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Apps from Deployments",
			"processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
			"type": "Application",
			"filter": {
				"exactType": "Deployment",
				"onRead": "${lx.factsheet.name==data.name && lx.tags.size()>0 && lx.tags[0].name=='Public Cloud'}"
			},
			"identifier": {
				"external": {
					"id": {
						"expr": "${content.id}"
					},
					"type": {
						"expr": "externalId"
					}
				}
			},
			"updates": [
				{
					"key": {
						"expr": "name"
					},
					"values": [
						{
							"expr": "${lx.factsheet.name} (Cloud)"
						}
					]
				}
			],
			"logLevel": "debug",
			"read": {
				"fields": [
					"name"
				],
			  "tags": {
			    "groups": [
		    		"Cloud Transformation"
	    		]
    		}
			}
		}
	]
}

The above example adds " (Cloud)" to the name of a Fact Sheet if current the name is exactly as defined in the data object and a tag "Public Cloud" in a tag group "Cloud Transformation" is set on the Fact Sheet.

❗️

onRead is available for inbound processors only

The onRead filter is available for inbound processors. The outbound processor will ignore setting this filter configuration.
Functionality is available same way outbound however as it is possible to use read content in the "advanced" filter like e.g. "filter": {"advanced": "${lx.relations.size()>0}"}, for cases where you only want to export if any requested relation was found

Example Mapping Use Cases

📘

Order of RegEx execution

Using the replace regEx will allow to modify the output after applying the match regEx

Loading large Data sets

Depending on the source of the incoming data, LDIF files can be very large. The Integration API may not accept LDIF larger than 50MB. If your file is bigger, it needs to be provided as a URL reference on an Azure Blob storage.
The configuration can be added to the Processor configuration part:

{
  "dataProvider": {
    "url": "${header.customFields.url}"
  }, 
  "processors": [
  ]
}

The value can be configured fix or as in the example passed in as part of the custom fields information in the LDIF. Please ensure to not send the "content" section in case you want to read from Azure. If content is part of the API call, this content will be used instead of the content in the Azure storage.
The URL needs to contain the path to the blob storage entry plus the Azure SAS token. See Azure documentation for details. https://docs.microsoft.com/bs-latn-ba/azure/storage/blobs/storage-blob-user-delegation-sas-create-cli

📘

No support for IP while listing

Please note that we process from our Azure infrastructure where IP addresses can dynamically change. Reading data from a URL does only work if no IP white listing is mandatory. Instead SaS tokens with limited ttl can be used.

Long running Integration API runs

The default behaviour of Integration API is to execute all changes with the user that is provided by the API client when logging in. E.g. the history of factsheets will show this user as the one executing the changes as if the user logged into the LeanIX UI and did the changes manually. This is easy to understand and communicate. In some situations however, it may not be sufficient.
Use Cases may be:

1. The process takes more than 60 minutes
2. All changes, the API does are caused by different login users, but should rather be shown to the user as a change the Integration API did than showing all the different API users.

In order to support long running inbound or outbound processes, an API Token may be provided in the configuration. This will be used instead of the access token that comes with the call from the API (as it will expire after one hour and does not contain any refresh token to grant access to the workspace data for more than 60 minutes)

{
  "credentials": { 
    "apiToken": "..." 
  }, 
  "processors": [
  ]
}

As an alternative, the Integration API can be configured to use a "Technical User" for accessing the LeanIX Pathfinder backend. This user will be automatically created in the workspace by the API if it is not existing. It can then be reconfigured by the administrator.
To use the Technical Users, please add the following section

{
  "credentials": { 
    "useTechnicalUser": true 
  }, 
  "processors": [
  ]
}

Search Based Matching of Fact Sheets

When using the "search" based identification of the Fact Sheet that are supposed to be updated by the incoming data object, then the section may contain a section to limit the scope of searched Fact Sheets and an expression filtering the Fact Sheets that should be updated.
When configuring an inboundFactsheet processor, a key "search" is now allowed. the value of this key is an object as defined in the example below. One or more Fact Sheets may be identified by the search and be updated based on the same data object in the LDIF.
The search works in two steps:

1. The "scope" defines a search against the pathfinder backend and limits the number of Fact Sheets to be matched. A valid scope can e.g. be created by using an outbound Integration API configuration and click on "set scope". Then the Scope can be copied from there.
2. After reducing the scope of Fact Sheets potentially in scope as good as possible with pathfinder filtering options, an additional JUEL expression is being executed to further narrow down the scope (key: "filter"). This is the far more costly process but allows much more flexibility identifying the right Fact Sheets. Admins should always try to limit the scope in phase 1 as far as possible to avoid long processing times. In the JUEL expression all fields defined in the "read" section can be used for filtering (e.g. ${lx.factsheet.description.startsWith('Autoupdate: ')}
3. All identified Fact Sheets will be processed as if they had been found by the processor one by one.

The "search" key can be used in conjunction with the "external" key. In this scenario, the Integration API first tries to find based on the external ID. If that fails, the search will be executed. If that fails as well, the integration API will try to create the Fact Sheet based on information in the "external" value. The last step can be avoided if creation is not allowed for the use case by adding an onRead filter and check for "lx.factsheet" not being null.
From a use case perspective, it allows to search for a fact sheet with a specific external id, if not existing then search for a fact sheet with e.g. a specific name and add the external id in case it is existing. OR create a new fact sheet in case a Fact Sheet with the name was not found.

The key "multipleMatchesAllowed" allows to define the API behaviour in case multiple Fact Sheets are matching the search criteria. Some use cases may only want to update if exactly one fact sheet was found (then the value will be set to "false"). By allowing multiple matches, bulk updates on multple Fact Sheets are possible. Default if not existing is "true"

The Below Processor will update all descriptions of Application Fact Sheets that have a tag "AsiaPacific" in the tag group "Region".

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Update all Cloud Apps",
			"processorDescription": "Updates all Apps with tag 'Cloud'",
			"type": "Application",
			"filter": {
				"exactType": "AppUpdate"
			},
			"identifier": {
				"search": {
					"scope": {
						"facetFilters": [
							{
								"facetKey": "FactSheetTypes",
								"operator": "OR",
								"keys": [
									"Application"
								]
							},
							{
								"facetKey": "${integration.tags.getTagGroupId('Region')}",
								"operator": "OR",
								"keys": [
									"${integration.tags.getTagId('Region','AsiaPacific')}"
								]
							}
						],
						"ids": [

						]
					},
					"filter": "${true}",
					"multipleMatchesAllowed": true
				}
			},
			"logLevel": "debug",
			"updates": [
				{
					"key": {
						"expr": "description"
					},
					"values": [
						{
							"expr": "External sync executed ${data.dateTime}"
						}
					]
				}
			]
		}
	]
}

You may start a test with this sample LDIF data

{
	"connectorType": "searchBasedScope",
	"connectorId": "searchBasedScope",
	"connectorVersion": "1.0.0",
	"lxVersion": "1.0.0",
	"description": "Updates external sync date",
	"processingDirection": "inbound",
	"processingMode": "partial",
	"customFields": {
	},
	"content": [
		{
			"type": "AppUpdate",
			"id": "apps",
			"data": {
				"dateTime": "06/08/2019"
			}
		}
	]
}

Extending the example is easy to e.g. only match update the Fact Sheets where the description already starts with a specific text, indicating that an automatic update was allowed:

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Update all Cloud Apps",
			"processorDescription": "Updates all Apps with tag 'Cloud'",
			"type": "Application",
			"filter": {
				"exactType": "AppUpdate"
			},
			"identifier": {
				"search": {
					"scope": {
						"facetFilters": [
							{
								"facetKey": "FactSheetTypes",
								"operator": "OR",
								"keys": [
									"Application"
								]
							},
							{
								"facetKey": "${integration.tags.getTagGroupId('Region')}",
								"operator": "OR",
								"keys": [
									"${integration.tags.getTagId('Region','AsiaPacific')}"
								]
							}
						],
						"ids": [

						]
					},
					"filter": "${lx.factsheet.description.startsWith('External sync')}",
					"multipleMatchesAllowed": true
				}
			},
			"logLevel": "debug",
			"read": {
				"fields": [
					"description"
				]
			},
			"updates": [
				{
					"key": {
						"expr": "description"
					},
					"values": [
						{
							"expr": "External sync executed ${data.dateTime}"
						}
					]
				}
			]
		}
	]
}

📘

Using ExternalIDs in search based scope (and in deletion scope)

Scope filters may not only filter for items by their internal ids (key "ids") but by external ids as well. To use this, the key "externalIds needs to be defined and contain an array of searched external ids. Each need to be prefixed with the name of the external id field and a slash. See example below

The Below example can even be dynamic and inject content from the LDIF custom fields:
"externalIds": ["${'externalId/'.concat(header.customFields,myExternalId)}"]

"scope": {					
					"facetFilters": [],
                    "externalIds": ["externalId/Ext-ID-0m0NiY6Z"]
				}

Using Bookmarks to define the search based scope

It is possible to not hardcode the search scope into the Integration API configuration but allow users with access to specific bookmarks to dynamically change the scope of Integration API runs by modifying the Bookmark in the Frontend of the application. In Integration API the ID or name of a Bookmark can be configured to be used to set the scope for search based scoping as shown in the below example:

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Apps from Deployments",
			"processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
			"type": "Application",
			"filter": {
				"exactType": "Deployment"
			},
			"identifier": {
					"search": {
					"filter": "${true}",
					"multipleMatchesAllowed": true,
					"scopeFromBookmark": "${integration.bookmarks.getBookmarkId('book-1')}"
				}
			},
			"updates": [
                {
					"key": {
						"expr": "description"
					},
					"values": [
						{
							"expr": "bookmark name: '${bookmark.name}' AND id=${integration.bookmarks.getBookmarkId('book-1')}"
						}
					]
				}                
			]
		}
	]
}

In case, a bookmark is used, the whole bookmark object is available to be used in JUEL expressions in the processor. The above example shows this by accessing the bookmark name in the updates section.

📘

Find out details about Bookmarks

You may output the bookmark object into the description field to inspect the structure and available information in a Bookmark. Use the "test run" mode to not alter any Fact Sheets.

Advanced information about search result when using search based matching

In case Integration API iterates over a search result, two variables can be used in all JUEL expressions: search.resultSize (indicating the total number of items we iterate over in the processor) and search.resultIndex (number of current item being iterated)

Read Only Fact Sheet Processor

In cases, you want to collect information from the resulting Fact Sheets into variables, you can enable the "readonly" mode on the inboundFactsheet processor. In the example below, all entries of the "releases" field of each application are collected into a variable releases. Results could then be used in a next run. Using the "filter": "${myExpression}", the set of fact sheet values collected can be even more narrowed down according to the use case. This feature comes handy to save processing time as the processor does not need to prepare any write operation.

{
			"processorType": "inboundFactSheet",
			"processorName": "Process variables with Search Scope",
			"processorDescription": "Collect deploymentMaturity ",
			"type": "Application",
			"filter": {
				"exactType": "Deployment"
			},
			"identifier": {
				"search": {
					"scope": {
						"ids": [],
						"facetFilters": [
							{
								"keys": [
									"Application"
								],
								"facetKey": "FactSheetTypes",
								"operator": "OR"
							}
						]
					}
				}
			},
			"run": 0,
			"enabled": true,
			"variables": [
				{
					"key": "releases",
					"value": "${lx.release}"
				}
			],
			"logLevel": "debug",
			"readOnly": true
		}

Accessing the "level" of a Fact Sheet in hierarchy

It is easily possible to access the hierarchy level of a Fact Sheet just by reading the field "level" provided by the pathfinder backend. This information can be used to filter for certain hierarchy levels or do calculations.
In the below example, it is used to filter for Level 2 Project Fact Sheets for export. This could of course have been done by just applying the restriction to the pathfinder scope query. The below is just to show case and allow extending for more advanced filtering

{
	"scope": {
		"facetFilters": [
			{
				"facetKey": "FactSheetTypes",
				"operator": "OR",
				"keys": [
					"Project"
				]
			}
		],
		"ids": []
	},
	"processors": [
		{
			"processorType": "outboundFactSheet",
			"processorName": "Export Projects L2",
			"processorDescription": "Exports only Level 2",
			"enabled": true,
			"filter": {
				"advanced": "${lx.factsheet.level==2}"
			},
			"fields": [
				"name",
				"level"
			],
			"output": [
				{
					"key": {
						"expr": "content.id"
					},
					"values": [
						{
							"expr": "${lx.factsheet.id}"
						}
					]
				},
				{
					"key": {
						"expr": "content.type"
					},
					"values": [
						{
							"expr": "project}"
						}
					]
				},
				{
					"key": {
						"expr": "name"
					},
					"values": [
						{
							"expr": "${lx.factsheet.name}"
						}
					]
				},
				{
					"key": {
						"expr": "level"
					},
					"values": [
						{
							"expr": "${lx.factsheet.level}"
						}
					]
				}
			]
		}
	]
}