# Transformer Configuration for Master Data Imports

This guide describes how to view and adjust the **import configuration** (Transformer Configuration) of a master data list in the [Master Data Hub](/en/docs/master-data/master-data-hub.md). It determines **which file** is read from the SFTP server, **how it is read** (encoding, delimiter, …), and **how the columns of your file** are mapped to the target schema expected by Workist. The same configuration applies to files you upload in the Workbench or deliver via the API.

Each master data list (articles, customers, customer contacts, addresses, unit conversions, framework contracts, orders, offers) has **its own** configuration.

---

## Prerequisites

- A Workbench account with the **Manager**, **IT** or **Admin** role (see [Users and roles](/en/docs/administration/users-and-roles.md)).
- A master data list in the Master Data Hub. [Creating a list](/en/docs/master-data/master-data-hub.md#liste-anlegen) describes how to create one.
- At least one sample file whose column names you know — otherwise you cannot map them to the target schema.

:::note Custom import scripts
For some lists, Workist sets up a custom import script, e.g. when several files are merged into one list. In that case, only the file name patterns and the file reader parameters take effect. Do not save column mappings in that case: the import fails otherwise.
:::

---

## Step 1 — Open the configuration

Open the list in the [Master Data Hub](/en/docs/master-data/master-data-hub.md) and switch to the **Configuration** tab. You can also get there from a channel:

- via the channel menu → **Manage master data** and the gear (**Manage lookup**) in the list's row, or
- via the channel configuration → **Master Data** and the gear next to the list's field.

Both gears open the list in a new tab.

---

## Step 2 — Page layout

The tab consists of three parts:

1. The **Import configuration** card with the sections **File Name Patterns**, **File Reader Parameters** and **Column Mappings**.
2. The **Dry Run** card, which tests the configuration without importing anything (see [Step 7](#schritt-7)).
3. A bar at the bottom that shows whether there are **Unsaved changes**. It contains the **Discard** and **Save** buttons.

---

## Step 3 — File name patterns

In the **File Name Patterns** section, you define which file in your SFTP folder `/export` counts as input for this list.

- **Fixed file name**, e.g. `customer_article_list.csv`.
- **Wildcard** `*`, e.g. `articles_*_full.csv`.
- **Date and time placeholders** `%Y`, `%m`, `%d`, `%H`, `%M` and `%S`, e.g. `%Y%m%d_Artikel.csv` for `20240315_Artikel.csv`. Wildcards and date placeholders cannot be combined in one pattern.
- **Regular expression** with the prefix `re:`, e.g. `re:Artikel_[0-9]+\.csv`.

If several files match the pattern, Workist imports the newest one. If a file matches the patterns of several lists, Workist imports it into each of these lists.

Enter **one** pattern per list. **Add pattern** lets you add further patterns, but the standard processing imports only one file per list; with several patterns, the import fails. Only a custom import script from Workist merges several files.

For files you upload in the Workbench or deliver via the API, the file name does not matter.

:::tip
Use patterns that are as restrictive as possible. A pattern that is too broad, such as `*.csv`, may cause non-article files to be mistakenly interpreted as an article list.
:::

---

## Step 4 — File reader parameters

In the **File Reader Parameters** section, you configure how Workist reads your file. Each parameter is initially **Not set**; the default value then applies.

| Parameter | Meaning | Example |
| --- | --- | --- |
| **Delimiter** | Column separator in CSV files. The default is the comma. Workist does not detect the delimiter automatically during the import: if the file uses a different delimiter than set, the import fails with **Unexpected delimiter**. | `;` or `,` |
| **Encoding** | Character set of the file. The default is UTF-8. For CSV files from Excel saved as "UTF-8 with BOM", select `utf-8-sig`. | `utf-8`, `utf-8-sig`, `utf-16`, `latin-1`, `cp1252`, `ascii` |
| **Quote Character** | Character used to enclose values in the CSV file. The default is `"`. | `"` or `'` |
| **Escape Character** | Character used to escape the quote character within a value. | `\` |

The [dry run](/en/docs/master-data/master-data-hub.md#probelauf) suggests which values fit your file.

:::note
For **Excel files** (`.xls`, `.xlsx`), leave all parameters **Not set**. Workist detects the format from the file extension; parameters that are set make the import of Excel files fail.
:::

---

## Step 5 — Column mappings

In the **Column Mappings** section, you map the columns of your input file to the columns of the Workist target schema.

Each row consists of two fields:

- **Column Name** — the target column, selected from the columns of the master data type (see [Target columns per master data type](#zielspalten)).
- **Column Expression** — where the value comes from. Five types of expression are possible:

  | Expression | Meaning |
  | --- | --- |
  | `Artikelnummer` | **Direct column reference** — the value is taken 1:1 from the column of the same name. |
  | `{Bezeichnung1} {Bezeichnung2}` | **Template expression** — placeholders in curly braces are replaced by column values. |
  | `'STK'` | **String literal** — the value is hardcoded (e.g. a constant unit). |
  | `PASS` | **Keep if present** — the column of the same name is passed through; if it is not there, the mapping is ignored. |
  | *(empty)* | **No mapping** — the row is removed on save. |

Example for an article list:

| Column Name | Column Expression |
| --- | --- |
| `article_id1` * | `Artikelnummer` |
| `description` * | `{Bezeichnung_DE} {Farbe}` |
| `gross_price` | `Listenpreis` |
| `order_units` | `Bestelleinheit` |

What to keep in mind:

- **Only mapped columns are imported.** As soon as a mapping is saved, Workist only takes over the mapped columns and ignores all other columns of the file. Without mappings, Workist takes over the file's columns unchanged. They must then already be named like the Workist columns.
- **Rows marked with `*`** (**Required column**) are always shown in the form for each master data type; you cannot remove them. Leave the column expression empty if you don't need a mapping for such a column. The import does not check whether these columns are filled. Which columns Workist actually needs for recognition is described on the pages of the individual [master data types](/en/docs/master-data.md).
- **`partition_id`** is prefilled with `'main'` in the form. `main` means: the row applies to all customers (see [Assignment to customers](/en/docs/master-data.md#partition-id)). If your file contains a column with the customer number, replace `'main'` with its name.
- **Not mapped** rows are added by the dry run for columns of your file that don't correspond to a Workist column. Select a target column for them; otherwise Workist removes the row on save.
- **An orange `!`** (**Column not part of the target schema**) marks a target column that does not exist for this master data type. Such a mapping makes the import fail; remove the row.

Use **Add column mapping** to add further mappings until all columns of the target schema are used. Each optional mapping can be removed again via the trash icon.

---

## Step 6 — "Raw JSON" tab for advanced fields

Via the **Raw JSON** tab, you see the complete Transformer Configuration as a JSON document and can edit it directly.

In the JSON tab, fields that are not shown in the form are available in particular:

- `apply_common_functions` — a list of predefined transformations that are applied to the file in the specified order, e.g.:
  - `query` — filter out rows based on a condition (`"query": "source_unit != target_unit"`).
  - `strip` — remove leading and trailing whitespace.
  - `drop_column` — remove a column from the result.
  - `remove_duplicated_rows` — remove duplicate rows.
  - `remove_leading_zeros` — strip leading zeros from a column (common for article numbers).
  - `set_default_order_unit` / `set_default_order_unit_to_sales_unit` / `ensure_sales_unit_in_order_unit` — standardize order units.
  - `eval_expression` — evaluate an expression to compute a new column value (e.g. a composite customer description).
  - `add_incremental_id` — add a sequential ID as a column.
  - `filter_by_length` / `detect_chars` — filter rows based on the length or a pattern in a column.
- `replace_nan` — replace empty values of a column with a default value.
- `query` — global filter condition that is evaluated immediately after reading the file.

Example JSON for an article list with mandatory handling:

```json
{
  "file_name_patterns": ["customer_article_list.csv"],
  "file_reader_kwargs": {
    "encoding": "utf-8",
    "delimiter": ";"
  },
  "columns": {
    "article_id1": "Artikelnummer",
    "description": "{Bezeichnung_DE} {Farbe}",
    "partition_id": "'main'",
    "gross_price": "Listenpreis",
    "order_units": "Bestelleinheit",
    "sales_units": "Verkaufseinheit"
  },
  "apply_common_functions": [
    { "strip": "description" },
    { "remove_leading_zeros": "article_id1" },
    { "remove_duplicated_rows": ["article_id1"] }
  ]
}
```

As soon as you change something in the JSON, a red notice appears at the bottom if the JSON is not syntactically valid. Use **Format JSON** to indent and tidy the text without changing its content.

:::warning
The column names on the **left** side of `"columns"` must come from the target schema of the respective lookup type — otherwise the configuration is rejected on save.
:::

---

## Step 7 — Test and save {#schritt-7}

1. Test the configuration with a [dry run](/en/docs/master-data/master-data-hub.md#probelauf). It also uses unsaved changes and shows how Workist transforms your file. A mapping to an unknown target column or a column missing in the file shows up here, before a real import fails.
2. Click **Save** in the bar at the bottom. Workist then reloads the saved configuration. **Discard** removes unsaved changes.

:::info When does the new configuration take effect?
From the **next** import. Workist does not automatically reprocess data that has already been imported. To apply the new configuration immediately, start an [import from SFTP](/en/docs/master-data/master-data-hub.md#import-aus-sftp) in the Master Data Hub, upload the file again, or deliver it again via SFTP. After a configuration change, Workist also reprocesses an unchanged file.
:::

---

## Target columns per master data type {#zielspalten}

Which columns are available in the **Column Name** field depends on the master data type. What each column means and what Workist needs it for is described on the linked page.

| Master data type | Always shown in the form (`*`) | Other target columns |
| --- | --- | --- |
| [Articles](/en/docs/master-data/articles.md) | `article_id1`, `description` | `partition_id`, `article_id2`, `article_id3`, `article_status`, `gross_price`, `order_units`, `sales_units`, `sales_organisation`, `variants`, `extra_data` |
| [Customers](/en/docs/master-data/customers.md) | `client_id1` | `client_id2`, `client_id3`, `company_name`, `address1`, `street_name`, `street_number`, `address2`, `zip_code`, `city`, `country_subdivision`, `country`, `po_box`, `misc`, `client_type`, `client_contact_person1`, `domain`, `iln`, `vat`, `iban`, `swift`, `locale_numeric`, `locale`, `internal_note`, `partition_id`, `extra_data` |
| [Customer contacts](/en/docs/master-data/customer-contacts.md) | `contact_id`, `first_name`, `last_name`, `partition_id` | `phone`, `mobile_phone`, `email`, `extra_data` |
| [Address](/en/docs/master-data/delivery-and-invoice-recipients.md) | `address_id1` | `partition_id`, `address_id2`, `address_id3`, `name`, `address1`, `street_name`, `street_number`, `address2`, `zip_code`, `city`, `country_subdivision`, `country`, `po_box`, `misc`, `contact_person`, `phone`, `extra_data` |
| [Unit conversions](/en/docs/master-data/unit-conversion.md) | `source_unit`, `target_unit`, `conversion_quotient` | `partition_id`, `article_id`, `extra_data` |
| [Framework contracts](/en/docs/master-data/framework-agreements.md) | `partition_id`, `contract_id1` | `client_id`, `client_name`, `contract_id2`, `contract_name`, `extra_data` |
| [Orders](/en/docs/master-data/orders-to-be-matched.md) | `order_number` | `partition_id`, `external_reference`, `order_date`, `delivery_address`, `receiver_number`, `order_total_price`, `iban`, `swift`, `account`, `payment_terms_type`, `payment_terms_discount`, `payment_terms_due_days`, `position_number`, `article_id1`, `article_id2`, `article_id3`, `article_description`, `article_price`, `price_unit`, `quantity`, `unit`, `total_price`, `delivery_date`, `extra_data` |
| [Offers](/en/docs/master-data/quotes.md) | `offer_number` | `partition_id`, `valid_from`, `valid_until`, `position_number`, `article_id1`, `article_id2`, `article_id3`, `article_description`, `quantity`, `unit` |

The form always shows the columns marked with `*`; they are not mandatory in the sense of an import check. For articles, the selection also contains `description_boosted` and the `order_unit_conversion_*` columns. These are internal or deprecated; for units, use the [unit conversion](/en/docs/master-data/unit-conversion.md). Only use the target columns from this table.

---

## Help and support

If the import continues to fail after a change, or the desired values do not end up in the Workist schema, contact your point of contact at Workist and have the following information ready:

- the **name of the list** in the Master Data Hub and the **master data type** (e.g. "Articles"),
- the **time** of the failed import (column **Imported at**),
- the **input file** of the import (linked in the import history),
- the **error message** of the import or the result of a dry run.