Sourcing Engine


The purpose of this document is to explain how the sourcing process works and the different sourcing rules that are configurable in the Admin interface.

Main features

Before describing the steps of the process, we identify the main features of the sourcing engine:

  • Receive sourcing requests via service bus
  • Supports mode:direct or mode:batch - configurable at the client or order level.
  • Process wave of orders for a specific merchant
  • Fetches and understands store/aggregate/source relationship
  • Order errors across the system are centrally triaged and dealt with
  • Orders maintain status records as they traverse the system and can be queried at any time

Sourcing Request Scoring in the Sourcing Engine

We will explain how the sourcing engine works. The sourcing process is divided in the following 4 steps:

Alt Text

Sort batch

Either by client configuration or by request-specific indication, the Sourcing Engine processes sourcing requests in one of two modes: direct or batch. Requests that should be processed in the direct mode are immediately sourced. Requests that should be processed in the batch mode are batched together with all other “batch” sourcing requests and processed together at client-configurable intervals.

Note that the In store pickup orders are always “forced” to be sourced directly (regardless of the defined configuration) to ensure the fastest preparation of the order for customer collection.

Batching allows the sourcing engine to better optimize the use of source stock across a range of requests. This allows you to ensure priority options are more likely to be sourced successfully and that overall shipments and back-orders are minimized.

The sourcing engine’s sorting is currently based on three criteria: age, shipping method, and request size (line count). If a client wishes to truly minimize splits, they might sort first by request size, then shipping method, and finally age. If, instead, a client wishes to ensure orders are shipped as soon as possible, they might sort first by age, size, and shipping method.

Note: this configuration is available in the configuration portal, which is not accessible externally yet. Please contact Magento to change this configuration. Alt Text

Option generation

The sourcing engine searches for the best possible sourcing option. Conceptually, the sourcing engine builds a “search tree” to explore all possible ways to source an order. At each successive level in this tree, the sourcing engine explores all the sources available for the next successive line in the order.

In this way, the sourcing engine generates all viable (by the customer restrictions) sourcing options. It then scores each option and retains the sourcing option with the best score. This is the “winner” and is returned to OMS as the sourcing response along with the other 10 best scored options for reporting purposes (so that an admin can see the best scored options generated for each order).

Option Scoring

The sourcing engine implements two types of rules: filter rules and scoring rules.

The “filter rules” are strict yes/no votes on a given sourcing option providing restrictions on what a customer will allow. Examples of filter rules are “Max Splits” and “Source provides required services”. All rules (filter and scoring alike) can be enabled and disabled by client configuration. During its preparation phase, the sourcing engine queries the configuration system and request the client’s configuration for each rule. This configuration can include rule-specific information such as the exact limit for the “maximum number of shipment” filter rule. But they all also contain an “enabled” flag by which the client can choose to use or not use any given rule.

Scoring rules, on the other hand, evaluate sourcing options based on customer criteria (if there are any). Each enabled rule provides a score for a given sourcing option. Scores range from 10.0 (best) to 0.0 (worst) and the set of scores across all options are scaled to this range to ensure that the best options receive 10.0 and the worst receive 0.0. To accomplish this, the sourcing engine scores in two phases. In the first phase, it looks at all the generated sourcing options and allows each rule to gather any “meta” information it will need to score. Often, this “meta” information is used to scale the scoring results to the range 0.0-10.0.

Scaling is important to ensure that rules can be fairly evaluated against each other. In most cases, a customer wishes to score options across multiple rules/criteria. For instance, while minimizing shipments in a single order maybe important, the customer might also wish to prioritize certain warehouses over others. The relative importance of these rules is rarely equal and, to handle these situations, we provide a mechanism by which the customer can give a weight to each rule. These weight values are, for practical purposes, boundless meaning you really could attach a weighting of 10,000,000 to one rule and 1 to another. This weighting system is providing a lot of flexibility in controlling sourcing decisions.

Note: this configuration is available in the configuration portal, which is not accessible externally yet. Please contact Magento to change this configuration. Sourcing scoring rules

Available Sourcing Rules

  • Configurable maximum number of splits allowed for each order
  • Weighted rule: Minimize splits for a single order
  • Weighted rule: Prioritize sources by rankings
  • Weighted rule: Prioritize sources by stock availability
  • Weighted rule: Prioritize sources by distance
  • Weighted rule: Preference for sources in the same shipping zone as the delivery address
  • Rule Weight definition for the range of 5 values (Lowest, Low, Medium, High, Highest)
  • Prevent splits across bundled products
  • Flag allowing the engine to split the bundle after reaching a threshold of failed attempts to source the request. Note that the bundle must still be completely sourced in order to successfully source the request.
  • Prevent sourcing in case some of the lines don’t have stock, and hold back the entire order as backorder
  • Flag allowing the engine to ignore the “Prevent partial sourcing rule” after reaching a threshold of failed attempts to source the request.
  • Capabilities: Source lines to sources that provide required service
  • Capabilities: Source lines to sources that provide required shipping method (additional paramenter is available to make standard shipping to be always assumed true)
  • Filter shipping zones by defining a zone mapping and assign each source to a specific zone. Orders will be sourced only to sources that are in the same shipping zone as the defined shipping address (based on two configurable criterias state or country). In case a source is not part of any defined zone can be used as a fallback.
  • Allow for configuration of maximum orders to be sourced from a given source. Each source will define the max number of orders that can be allocated on 1 day. The rule applies to all the days of the year. The definition of the max orders is done via the source create/update messages under the capabilities section, and can be managed via the source page in Admin. [See Specifications: magento.inventory.source_management].

Configure allocation waves

For clients configured to be sourcing in batch mode there is a much better chance to source orders in a optimal way, according to the defined rules. Via configuration it’s possible to define when the batch process should be run (only once per hour) and to indicate which source will be considered active in each one of the executed batches. At the moment, this configuration is only defined in UTC. A source will inherit it’s wave schedule, but can override to add or remove waves. A source with no times selected becomes inactive.

The wave schedule is also used to process the backorder queue, each wave will validate if the stock for the queued orders is available and in that case will source the backorders.

A configuration will define the sorting to be used to process batched orders based on the following paramenters: size, age and shipping method.

Allocation waves are used for 2 different purposes:

  • Client sourcing in mode batch will be batching all newly created orders with the orders sitting in the exception queue and will run the sourcing logic at the time defined. For these clients it will be possible to define which source should be considered active in a specific wave (i.e. during one wave we might want several sources not to be considered and use other options available) > see lower how to disable sources for a specific wave
  • Clients sourcing in mode direct are still using the waves to process orders in the exception queue. In this case all sources are always assumed to be active for each one of the wave (therefore if the configuration is overwritten at the source level it will be ignored)

Allocation waves times can be configured either at root level or at aggregate level. When defined at root level this implies that all existing sources associated to the aggregate will be considered as available during each one of the waves. In order to disable a specific source for a specific time, the configuration key will have to be removed under the specific source node as follows:

  • in the Scope dropdown select the source
  • navigate to the configuration Sourcing > Active Wave Times
  • uncheck the input field on the right saying DEFAULT
  • remove the unwated wave by clicking the trash can icon

Remove Wave

IMPORTANT: if root doesn’t have any wave configured and those are only configured at the aggregate level, this will mean that none of the sources will automatically inherit the setting, therefore assumed to be switched off. In this case each one of the sources will have to define the exact same waves that are existing for the aggregate. In order to optimize the creation of configs, those can be defined in the MetaSource leave so that will be inherited by all children sources.

Force sourcing decision at order line

Is also possible to define at order line level the source that should be used to force the sourcing. In case the source_location is defined then other available options for sourcing will be ignored and the lines will always be forced to the defined source. Note: in case stock is not available in the source_location the line will be backordered untill stock becomes available in this source.

Key Value Message Description Sample
Force Source Location (optional) magento.sales.order_management.create With the order creation is possible to pre-define the source location that will be shipping the items. Can be used for both home delivery or ISPU orders (this last only in case ship to store feature is enabled) STS sample 1

Integration order flow

Please, find below the order flow where you can see the four steps explained previously.

Alt Text

Alt Text