Magento Integration Bus

Introduction

The Magento Order Management Integration Bus is an API gateway which provides a common communication layer for different applications to interact with Magento Order Management. It provides basic service discovery capabilities, authorization and authentication as well as synchronous, asynchronous and broadcast communication mechanisms.

Magento Integration Bus plays the role of a centralized API gateway and messaging broker which receives requests from everywhere and routes them to corresponding applications. Applications within Magento Integration Bus are addressed using unique identifiers.

Every application registers itself in Magento Integration Bus poviding basic information about itself. Things like application unique identifier, its URL, topics it wants to subscribe to, secret, etc. This process helps Magento Integration Bus to learn about new integration and gives instructions about where and how messages should be delivered.

Transport protocol

Magento Integration Bus uses JSONRPC 2.0 over HTTP as simple and widely spread set of protocols. Although, please keep in mind there are few exceptions in how Magento Integration Bus handles JSONRPC calls.

  1. Batch requests are not supported at the moment
  2. Magento Integration Bus only supports params encoded as an object: "params":{"foo":"bar"} - works, but "params":["bar"] - does not work.
  3. At the moment only HTTP 1.0 and HTTP 1.1 are supported

For any question regarding JSONRPC, consult specification published at http://www.jsonrpc.org/specification.

Communication types

Magento Integration Bus was designed to provide different mechanisms for applications to communicate among themselves. At the moment there are three communication types: synchronous, asynchronous and broadcast.

Synchronous (one-to-one)

Synchronous communication type is when one application makes a call to another and waits for a response. In this case Magento Integration Bus plays the role of a proxy which simply forwards request and response between two applications.

Asynchronous (one-to-one)

Asynchronous communication type is when one application makes a call to another without waiting for a response. In this case the Magento Integration Bus just acknowledges receipt of the request and delivers message separately, maybe with a little delay. Magento Integration Bus will also re-try to deliver a message in case of an error. You can think of asynchronous communication as if your application would delegate a call to the Magento Integration Bus. Once Magento Integration Bus took responsibility to deliver the message (acknowledge it), your application can safely assume that the message will be eventually delivered.

Broadcast (one-to-many)

Broadcast communication type is when one application broadcast a message to zero or many other applications depending on their interest in the message topic. Every application can provide a list of topics it wants to subscribe to in the registration call. Broadcast is similar to the asynchronous call: the Magento Integration Bus will simply acknowledge that broadcast is received and queued. Then, the message will be delivered and retried in case of an error.

Endpoints

Magento Integration Bus provides multiple endpoints for different applications and communication types. Magento Integration Bus base URL looks something like https://mib.mom.magento.com/luma/.

URL https://mib.mom.magento.com/luma/ is just an example. Please, consult Magento Technical Support team for exact URL.

Base endpoint /

Base endpoint provides an API to interact with the Magento Integration Bus itself. This is the endpoint you should use for making registration and discovery calls. A list of all available methods available can be found here.

Example for base endpoint: https://mib.mom.magento.com/luma/.

Synchronous endpoint /remote/{app-id}

Synchronous endpoint proxy every call to given application. For example, call to endpoint /remote/hello-app will instruct Magento Integration Bus to lookup for URL for application with ID hello-app. If such application is registered, Magento Integration Bus will make a RPC call with exactly same parameters to this URL and return response as it is back.

Example for synchronous endpoint: https://mib.mom.magento.com/luma/remote/oms.

Asynchronous endpoint /delegate/{app-id}

Asynchronous endpoint (or delegate endpoint) drops every call into a queue and immediately acknowledges the request. For example, a call to endpoint /delegate/hello-app will instruct Magento Integration Bus to lookup for URL for application with ID hello-app. If such application is registered, Magento Integration Bus will drop the message into an internal queue, and reply to the original request with null, meaning request was accepted and persisted.

Example for asynchronous endpoint: https://mib.mom.magento.com/luma/delegate/oms.

Broadcast endpoint /events

Broadcast endpoint broadcasts every RPC call received to all subscribers. For example, RPC to method magento.service_bus.explained will be delivered to any service subscribed to the topic which match method name.

Example for broadcast endpoint: https://mib.mom.magento.com/luma/events.

Authorization

Magento Integration Bus uses OAuth2 to authorize all incoming HTTP requests. So, before making any calls to the Magento Integration Bus you need to receive an OAuth2 token. It’s valid for a limited amount of time, so you need to develop your application in a way to dynamically fetch newer token in case Access denied (error code: -32604) error is received.

Once you receive a token you should send it in Authorization header like so: Authorization: Bearer TOKEN-HERE. In examples below placedholder TOKEN is used to mark where OAuth2 token should be.

Fetching token

Before continue, please consult the Magento Technical Support team to receive your OAuth2 Server URL, Client ID and Client Secret.

It takes one HTTP call to receive an OAuth2 token:

curl -X POST https://auth-stg.bcn.magento.com/oauth/token \
 -F grant_type=client_credentials \
 -F client_id=CLIENT-ID \
 -F 'client_secret=CLIENT-SECRET'

Then, you will receive 2xx response with the following payload:

{"access_token":"QlhjQclLI0TSS8UcHms4pCLUugq2Aris654HknyH","token_type":"Bearer","expires_in":3600}

Your OAuth2 token is in the field named access_token.

Registration

Registration process, and API call meant to add or update information about an application connected to the Magento Integration Bus. In case you try to register an application with ID which already exists in the Magento Integration Bus it will override previously registered record.

Important! For each attempt to register a new application, a probe will be made against the registered URL using the OPTIONS HTTP verb. The server should reply with an empty body including the following header: X-Magento-Service-Bus: *. This process is called “URL probing” and should prevent the Magento Integration Bus from making calls to URLs which are not meant to be used as Magento Integration Bus endpoints.

Registration can be performed during application deployment process, or periodically by some scheduled process, for example every hour. You can also perform registration manually, althouh it’s preferred to automate this process.

All parameters for a registration call are described in API specifications, see “register” command.

Example

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.register",
    "params": {
        "id": "warehouse-integration-example",
        "url": "https://warehouse-integration.mom/api"
    }
}'

Unregistration

Please, use this API method with caution.

Unregistration process allows you to remove unused integration. It can be performed using “unregister” command.

Important! Unregistration will cause undelivered messages to be removed, and it won’t be possible to restore them later. Although, unregistration does not guarantee that all messages will be fully removed, if you register another application with the same name, some of the messages still maybe delivered within short period of time.

Example

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.unregister",
    "params": {
        "id": "warehouse-integration-example"
    }
}'

Discovery

Magento Integration Bus provides an API to fetch a list of all registered applications with their meta information. The response will contain the same information as was provided during registration, but secret. You can use the discovery call for debugging, to verify if your application was properly registered or to find out what applications are connected to the Magento Integration Bus.

All parameters for the discovery calls are described in API specifications, see “discover” query.

Example

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.discover"
}'

Synchronous communication

As mentioned above, the Magento Integration Bus provides a proxy functionality for synchronous calls.

Example

Assuming there is an application warehouse-integration-example registered with the same parameters as in the Registration example, we can send a synchronous call.

curl -X POST https://mib.mom.magento.com/luma/remote/warehouse-integration-example -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}'

Then, the Magento Integration Bus will make a POST request to URL to https://warehouse-integration.mom/api (URL used in registration call) with following payload:

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}

Then, the reply will be forwarded back.

Asynchronous communication

Asynchronous communication allows to leverage Magento Integration Bus’ retry capabilities and deliver message even if the application is not available at the moment of the call.

Example

Assuming there is application warehouse-integration-example registered with the same parameters as in the Registration example, we can send an asynchronous call.

curl -X POST https://mib.mom.magento.com/luma/delegate/warehouse-integration-example -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}'

You may notice that the call is exactly the same as for synchronous communication, only URL is different.

Then, the Magento Integration Bus will reply with an error, in case there is any or null meaning the request was accepted:

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}

Then, the Magento Integration Bus will make a POST request to URL to https://warehouse-integration.mom/api (URL used in the Registration call) with the following payload:

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}

Magento Integration Bus will put the message back to the queue in case an error occur at any moment (DNS failure, Network failure, non 2xx HTTP response, etc).

Broadcasting

Magento Integration Bus provides a simple publish/subscribe API for events. Any application can list topics it wants to receive and once a message with a given topic is published to the broadcasting endpoint, it will be delivered to a given API.

Example

In the API call below an application with ID event-subscriber subscribes for events: magento.foo and magento.bar.

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.register",
    "params": {
        "id": "event-subscriber",
        "url": "https://event-subscriber.mom/",
        "subscribes": [
            "magento.foo",
            "magento.bar"
        ]
    }
}'

Then, any other application could publish an event using the broadcast endpoint: https://mib.mom.magento.com/luma/events.

curl -X POST https://mib.mom.magento.com/luma/events -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.foo",
    "params": {
        "foo": "bar"
    }
}'

Then, Magento Integration Bus will reply with an error, in case there is any or null meaning the request was accepted:

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}

Then, the Magento Integration Bus will make a POST request to URL to https://event-subscriber.mom/ (URL used in the Registration call) with the following payload:

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.foo",
    "params": {
        "foo": "bar"
    }
}

In case of error the Magento Integration Bus will apply the same retry mechanism as for asynchronous communications.

Verify delivery

Once your application is registered as an endpoint for the Magento Integration Bus, it’ll accept any payload sent to the URL you configured. For security reasons, you probably want to limit requests to those coming from Magento Integration Bus. There are a few ways to do this - for example, you could opt to whitelist requests from certain IP address - but a far easier method is to set up a secret token and validate incoming requests.

Setting up secret

Magento Integration Bus uses a signature with pre-shared secret. It means Magento Integration Bus will use a secret (a string provided during registration) to sign all requests to your endpoint, and your application would need to verify that this signature is correct.

You should provide a secret for your endpoint during its registration, for example like this:

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.register",
    "params": {
        "id": "warehouse-integration-example",
        "url": "https://warehouse-integration.mom/api",
        "secret": "foo"
    }
}'

It won’t be a good idea to hardcode the token in your application. Best option will be to generate it dynamically during registration and then store it in some persistant storage, so you can validate signature. Or, alternativelly, you can put it in some configuration file.

Validating requests

Sevice bus will add a special header X-Signature in every request to URLs registered with non-empty secret. Keep in mind, if a URL is registered with empty secret, the header will not be set.

Signature is SHA1 hash in HEX format prefixed with sha1=, for example: sha1=9f00d52ea157870e92a975070e277748230443a3. The hash is generated from the request body and a secret provided during registration.

Here is a basic PHP code which verifies the signature:

$secret = "foo";
$raw = file_get_contents("php://input"); // request body

$signature = isset($_SERVER['HTTP_X_SIGNATURE']) ? $_SERVER['HTTP_X_SIGNATURE'] : null;

if ($signature === null) {
    return false; // Request is not signed!
}

$hash = "sha1=".hash_hmac("sha1", $raw, $secret);
if ($hash != $signature) {
    return false; // Signature does not match
}

return true; // Signature is valid!

Error processing

Magento Integration Bus uses HTTP protocol in transport layer. You should handle cases when HTTP protocol gives error response (status code is different than 200) and process them accordingly. Normally error in HTTP protocol would mean misconfiguration or some connectivity issue.

HTTP Status Code examples:

HTTP Status Codes Description
200 Request was successfuly delivered. Your should process response body for JSON RPC response (which may indicate error).
401 OAuth token is not passed in the HTTP headers or is expired. Your application should try to receive a new token from OAuth server.
404 The URL does not exist. You should verify URL you are using to make calls.
500 Generic transport error. You should reach Magento support team for details.

Other errors should be handled according to HTTP specification.

Once you receive HTTP response with status code 200, it means there was no errors on transport layer, you should perceed to parse response body as JSON RPC response.

JSON RPC response may still indicate an error, check JSONRPC specification for more details.

Error codes used by Magento Integration Bus:

Code Description Will Be Retried
-32000 Generic internal error yes
-32700 Request parsing error, for example: invalid JSON no
-32600 Invalid request error no
-32601 Method not found no
-32602 Invalid request parameters no
-32603 Internal error yes
-32604 Access denied no
-31101 Fail to reach server (for proxied calls) yes
-31102 Invalid reply from the server (for proxied calls) yes
-31001 Probe failed (see registration section) no

Error notification for asynchronous messages

While sync messages will get an immediate response on whether the message was sucessfully processed, asynchronous messages have to follow another path due to the nature of such delayed processes.

An integration may choose to send a specifically crafted http header with each request made to the Magento API in order to be notified if a message fails. Such header will contain the name of the integration itself that needs notification (usually the same integration that sends the message).

X-Error-Reply-To: <integration-name>

Example

An asynchronous request is made directed at OMS setting up the X-Error-Reply-To header so integration mdc is notified of any particular error:

curl -X POST https://mib.mom.magento.com/luma/delegate/oms -H'X-Error-Reply-To: mdc' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.sales.order_management.create",
    "params": {
        "incorrect": "payload"
    }
}'

Inmediate response from the Magento API would be (as expected for all async requests):

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}

As expected, such message would fail upon arrival to OMS, and if the X-Error-Reply-To header is set, the Magento API will attempt to notify the registered application mdc of said failure via the following message:

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.common.error_management.notify",
    "params": {
        "topic": "magento.sales.order_management.create",
        "message": "Validation failed:\n [order] This value should not be blank.",
        "payload": "{\"incorrect\": \"payload\"}"
    }
}

Where the params structure will be:

  • topic: Original topic processing where error occurred
  • message: Error message
  • payload: Payload of message that triggered the error (as a string)

Retry strategy

Magento Integration Bus will retry to deliver messages which were not delivered or accepted by the server. The system will retry delivering messages until it receives a valid JSON RPC response from the server which is not a JSON RPC error response with retriable error codes (see error codes section above) or 0.

Examples, when OMS will retry to deliver a message:

  • DNS error while resolving destination URL
  • Connection refused error while connecting to remote host
  • HTTP Response with status code different than 200
  • JSON RPC error response with error code: -32000.

Example, when OMS will NOT retry to deliver a message:

  • JSON RPC result response
  • JSON RPC error response with error code: -32601

Alarm Email

In the event of service bus not being able to deliver a certain message to an integration, there’s an alarm system in place that checks hourly for non delivered messages and creates a report sent to a configured e-mail address.

Magento Commerce Connector Alarm Email Setup

For integrations that use the Magento Commerce Connector this alarm email can be configured in the env.php configuration file inside the serviceBus.labels[] key:

return [
    ...
    'serviceBus' => [
      'url' => 'http://sample-mcom-url:8080/LUMA/',
      'secure_endpoint' => true,      'labels' => [
        // Email address to send alarms to
        'magento.alarms_email' => 'some@email.com',
      ],
    ],
    ...
);

This parameter can include more than one email address if needed concatenating them by pipes (e.g. 'magento.alarms_email' => 'some@email.com|another@mail.com',

After updating said file, run bin/magento setup:upgrade to trigger registation again and update Magento OMS API with the new label being added.

Manual Registration

For custom made integrations which do not use the MDC Connector, the process is the same as described in the registration section, adding the magento.alarms_email configuration to the labels section.


{
    "jsonrpc":"2.0",
    "method": "magento.service_bus.remote.register",
    "id": 1,
    "params": {
        "id": "name-of-integration",
        "url": "http://url-of-remote-endpoint/",
        "contracts": [
            ...
        ],
        "subscribes": [
            ...
        ],
        "labels": {
            "magento.alarms_email": "some@email.com"
        }
    }
}

For multiple emails to be notified, concatenate them with pipes as before "magento.alarms_email": "some@email.com|another@email.com"