Sourcing Engine


The purpose of this document is to explain how the sourcing process works and the different sourcing rules that are configurable in the Admin interface.

Main features

Before describing the steps of the process, we identify the main features of the sourcing engine:

  • Receive sourcing requests via service bus
  • Supports mode:direct or mode:batch - configurable at the client or order level.
  • Process wave of orders for a specific merchant
  • Fetches and understands store/aggregate/source relationship
  • Order errors across the system are centrally triaged and dealt with
  • Orders maintain status records as they traverse the system and can be queried at any time

Sourcing Request Scoring in the Sourcing Engine

We will explain how the sourcing engine works. The sourcing process is divided in the following 4 steps:

Alt Text

Sort batch

Either by client configuration or by request-specific indication, the Sourcing Engine processes sourcing requests in one of two modes: direct or batch. Requests that should be processed in the direct mode are immediately sourced. Requests that should be processed in the batch mode are batched together with all other “batch” sourcing requests and processed together at client-configurable intervals.

Note that the In store pickup orders are always “forced” to be sourced directly (regardless of the defined configuration) to ensure the fastest preparation of the order for customer collection.

Batching allows the sourcing engine to better optimize the use of source stock across a range of requests. This allows you to ensure priority options are more likely to be sourced successfully and that overall shipments and back-orders are minimized.

The sourcing engine’s sorting is currently based on three criteria: age, shipping method, and request size (line count). If a client wishes to truly minimize splits, they might sort first by request size, then shipping method, and finally age. If, instead, a client wishes to ensure orders are shipped as soon as possible, they might sort first by age, size, and shipping method.

Note: this configuration is available the configuration portal, which is not accessible externally yet. Please contact Magento to change this configuration. Alt Text

Option generation

The sourcing engine searches for the best possible sourcing option. Conceptually, the sourcing engine builds a “search tree” to explore all possible ways to source an order. At each successive level in this tree, the sourcing engine explores all the sources available for the next successive line in the order.

In this way, the sourcing engine generates all viable (by the customer restrictions) sourcing options. It then scores each option and retains the sourcing option with the best score. This is the “winner” and is returned to OMS as the sourcing response along with the other 10 best scored options for reporting purposes (so that an admin can see the best scored options generated for each order).

Option Scoring

The sourcing engine implements two types of rules: filter rules and scoring rules.

The “filter rules” are strict yes/no votes on a given sourcing option providing restrictions on what a customer will allow. Examples of filter rules are “Max Splits” and “Source provides required services”. All rules (filter and scoring alike) can be enabled and disabled by client configuration. During its preparation phase, the sourcing engine queries the configuration system and request the client’s configuration for each rule. This configuration can include rule-specific information such as the exact limit for the “maximum number of shipment” filter rule. But they all also contain an “enabled” flag by which the client can choose to use or not use any given rule.

Scoring rules, on the other hand, evaluate sourcing options based on customer criteria (if there are any). Each enabled rule provides a score for a given sourcing option. Scores range from 10.0 (best) to 0.0 (worst) and the set of scores across all options are scaled to this range to ensure that the best options receive 10.0 and the worst receive 0.0. To accomplish this, the sourcing engine scores in two phases. In the first phase, it looks at all the generated sourcing options and allows each rule to gather any “meta” information it will need to score. Often, this “meta” information is used to scale the scoring results to the range 0.0-10.0.

Scaling is important to ensure that rules can be fairly evaluated against each other. In most cases, a customer wishes to score options across multiple rules/criteria. For instance, while minimizing shipments in a single order maybe important, the customer might also wish to prioritize certain warehouses over others. The relative importance of these rules is rarely equal and, to handle these situations, we provide a mechanism by which the customer can give a weight to each rule. These weight values are, for practical purposes, boundless meaning you really could attach a weighting of 10,000,000 to one rule and 1 to another. This weighting system is providing a lot of flexibility in controlling sourcing decisions.

Note: this configuration is available the configuration portal, which is not accessible externally yet. Please contact Magento to change this configuration. Sourcing scoring rules

Available Sourcing Rules

  • Sourcing schedule: Currently the time is hardcoded to the values 8:00, 13:00 and 15:00. The schedule will apply equally to all days of the year, including weekends and bank holidays. If a source doesn’t have any time selected, then automatically becomes inactive. At the moment, this configuration is defined in UTC.
  • Configurable maximum number of splits allowed for each order
  • Weighted rule: Minimize splits for a single order
  • Weighted rule: Prioritize sources by rankings
  • Weighted rule: Prioritize sources by stock availability
  • Weighted rule: Prioritize sources by distance
  • Rule Weight definition for the range of 5 values (Lowest, Low, Medium, High, Highest)
  • Prevent splits across bundled products
  • Flag allowing the engine to split the bundle after reaching a threshold of failed attempts to source the request. Note that the bundle must still be completely sourced in order to successfully source the request.
  • Prevent sourcing in case some of the lines don’t have stock, and hold back the entire order as backorder
  • Capabilities: Source lines to sources that provide required service
  • Capabilities: Source lines to sources that provide required shipping method (standard is assumed to be always true (enabled) so it is not checked if active)
  • Filter shipping zones by defining a zone mapping and assign each source to a specific zone. Orders will be sourced only to sources that are in the same shipping zone as the defined shipping address (based on two configurable criterias state or country). In case a source is not part of any defined zone can be used as a fallback.
  • Allow for configuration of maximum orders to be sourced from a given source. Each source will define the max number of orders that can be allocated on 1 day. The rule applies to all the days of the year. The definition of the max orders is done via the source create/update messages under the capabilities section, and can be managed via the source page in Admin. [See Specifications: magento.inventory.source_management].
  • Configurable sorting the batched orders based on: size, age and shipping method

The day time span is defined based on the timezone defined at source level and calculated from 12:00am to 11:59pm. In case a source don’t have a provided timezone the timespan is calculated on UTC

  • Minimize shipment for a batch of orders via heuristic
  • Older orders in a wave are sourced first
  • Larger orders in a wave are sourced first
  • Orders that were not sourced are added to the next wave

Force sourcing decision at order line

Is also possible to define at order line level the source that should be used to force the sourcing. In case the source_location is defined then other available options for sourcing will be ignored and the lines will always be forced to the defined source. Note: in case stock is not available in the source_location the line will be backordered untill stock becomes available in this source.

Key Value Message Description Sample
Force Source Location (optional) magento.sales.order_management.create With the order creation is possible to pre-define the source location that will be shipping the items. Can be used for both home delivery or ISPU orders (this last only in case ship to store feature is enabled) STS sample 1

Integration order flow

Please, find below the order flow where you can see the four steps explained previously.

Alt Text

Alt Text