In spite of having a decent documentation on webhooks, there were some major hurdles that I struggled to overcome when I set out to configure webhooks for our shopify app. It took an extensive perusal of Shopify community posts, independent blog posts and personal experimentation before I could get it to work. This post discusses the roadblocks overcome and workarounds devised before our shopify app was properly docked with it's mother ship.
There are 3 webhooks that every app submitted to shopify for review must register(in the App Setup section of the app in the partner portal) for GDPR compliance.
customers/redact- Called when a store owner requests deletion of data on behalf of a customer
customers/data- Called when a customer requests their data from a store owner
shop/redact- Called 48 hours after an app is uninstalled by the store
All the above are POST calls with different payloads for each. Detailed explanation of these webhooks can be found in the official documentation.
The partner portal will allow an app to be submitted for review without these URLs, but the app will get rejected as part of the review.
Apps can register these webhooks if they want to be notified when some events happen on a shop that has installed them. Shopify allows registering a callback URL against events that are supported by the platform. When the event occurs, shopify makes a POST call to the registered callback for that event with some payload as defined in the webhooks documentation.
While the webhooks documentation itself is self explanatory, there were 2 details that were skimmed over that made my experience with webhooks a nightmare
- HMAC validation
- Testing webhooks
To put in very simple terms, HMAC works like this. Shopify uses a secret key to generate an HMAC of the request body and sends it along with the request. The app that receives the callback should extract the request body and use the same secret key to generate an HMAC hash on its side. If the HMAC evaluated on the shopify side and passed along with the request matches with the HMAC generated for the request by the app, the request is an authentic request from Shopify. The security here relies on the principle that the secret key is known only to the app and the Shopify platform.
HMAC validation is optional and may be skipped. Nothing prevents a developer from creating an app that doesn't validate incoming requests. But keep in mind that this validation is done in order to ensure that the request received by the app has in fact originated from the shopify platform. Without this validation in place, you might end up creating an app that divulges information to an unauthorized third party. So embracing the dark side by choosing not to validate an incoming request is a serious security threat.
Shopify apps should do HMAC validation on incoming requests in 2 places
- In the install callback before doing access token exchange
- In all webhook callbacks
HMAC validation for NodeJS in the install flow is well documented for NodeJS with code in the tutorials. But for webhooks, it is a whole different story.
Getting hold of the shared secret
Shopify's install callback documentation makes a clear reference to an API secret key for HMAC validation. This key is visible upfront in the app's partner portal in the same name. So there was no confusion there. But the webhook verify documentation makes reference to a mystical shared secret in multiple places.
This shared secret added an hour or so to my ordeal of setting up a webhook. I scoured through a lot of documentation, the entire partner portal and community posts before I found in one of the posts that it is one and the same as the API secret key.
The shared secret key mentioned in the webhook verify documentation is one and the same as your API secret key in the partner portal of the app.
The first mistake I made was in assuming that the HMAC evaluation for webhooks is the exact same as the HMAC evaluation for install callback. Except for a very subtle difference(that took an excruciatingly long time to figure out), this assumption was correct.
In the install callback, shopify sends HMAC encoded as a hex value where as for webhook callbacks, HMAC comes as a base64 encoded value.
This point is casually mentioned in the shopify document about verifying webhooks but the code examples given there do not include NodeJS. The code evaluating HMAC for install callback needs to be tweaked a bit to accommodate it for webhooks
Input message for HMAC
The other detail that took me a lot of time to figure out was the message upon which the HMAC needs to be evaluated. Again this is very clearly mentioned in the install callback documentation, but with webhooks, I was looking at a cul-de-sac.
I read a couple of community posts claiming that the message to be hashed is the stringified JSON response body. Armed with confidence from these posts, I used Express body parser to parse the request body as a JSON and then stringifying this JSON to generate it's HMAC hash. Oh, how naive and wrong was I.
Then I came across posts suggesting that the raw request body straight off the network(without any intermediate parsing) is what gets the job done. This made sense to me as the generated hash can vary drastically even if a single character is off. That is when I set off on a quest, or rather a sub-quest, to retrieve the covetted raw request body. Soon enough, my sub-quest proved to be a daunting task in to execute in Express. Koa framework has a body parser that does this out of the box. But in Express body parser, it was either json or raw body. A solutions to getting both of them together seemed non-existent. Later, I found posts by some stackoverflow veterans, who had solved the problem some years back, by manually compiling the raw request from incoming chunks. But this solution(as shown below) did not work for me
Finally when I was on the verge of giving up on my sub-quest, my quest and my career as a web developer, I struck gold. Deliverance came from the Express body parser itself where I found the benign verify option in the json parsing section that naively advertised its willingness to provide me access to the raw request body.
The attempt I made with the verify option in body parser bore fruition and the below code was found residence in my codebase
After the raw body was extracted, the final code for evaluating HMAC for incoming webhook callbacks looked like this
HMAC validation for shopify webhook callbacks should be done on the raw request body straight off the network before passing through any body parsers
And so, with a panache of irony, the problems that were kicked off by a lacking verify documentation got resolved by the verify option of a package that resided in my app from the very beginning.
Once the webhooks were hooked up, next came the problem with testing them before submitting for review. Here too, there was not much resources that came to my aid other than bits and pieces in the community forum. The following are my learnings from getting this live. For purposes of the following discussion, I'm considering the following 4 webhooks that I worked with
customers/redact- Mandatory webhook
customers/data_request- Mandatory webhook
shop/redact- Mandatory webhook
app/uninstalled- Optional webhook
Callback URLs registered for Shopify webhooks muist be secure HTTPS links. So for development testing on localhost, ngrok is your best friend.
There are 2 conditions that needs to be satisfied for this webhook to be invoked
- The app should have been granted access for for order and/or customer related scopes mentioned in the access scopes doc
- There should be at least 1 customer object created in the shop admin portal
If the above conditions are satisfied, do the following to test this webhook
- Open any of the customers in the store admin page - https://mystore.myshopify.com/admin/customers
- Scroll to the bottom of the page and on the right side panel, will come the
- Click on
Request Customer Dataand raise a request
This will immediately raise a request to the callback that has been registered for
customers/data_request in the App Setup section of the app in partner portal.
app/uninstalled is an optional webhook which, if registered, will be triggered immediately after an app is uninstalled from the store. Unlike mandatory webhooks that can be configured in the partner portal, optional webhooks have to be registered by calling the create webhook API.
shop/redact is a mandatory webhook that will be invoked 48 hours after a store has uninstalled your app. Due to the long wait involved, testing this is not a practical option.
customers/redact is a mandatory webhook that must be configured in the partner portal. The conditions listed for
customers/data_request are applicable for this webhook also. If the conditions are met, this webhook can be tested(theoretically) by clicking on the
Erase personal data button in the customer privacy section. But testing this webhook is not a practical option given that the minimum duration that must pass before the callback URL gets invoked is 10 days and the maximum is 6 months. So either way, you are looking at a major experiment on the limits of mankind's patience.
That concludes my uphill journey to setup Shopify webhooks for our shopify app Product Labels by ModeMagic. Cheers!