PicoStack: Web API needing a token grant

Web APIs that follow the OAuth pattern need special treatment.

Besides needing a client id and client secret, API calls that do actual work require a bearer token, which is generated by a special call to the API.

So, using such an API is a two step process:

Given the client id and secret, request a token
Supplying the token, make the API call that does actual work

While one could do the two steps for each actual API call, it is better practice to store the token from the first call in an entity variable and then use it in each API call that does actual work.

We follow the pattern introduced in the Wrapper Opportunity post, of having a ruleset (herein called sdk) that wraps the actual web API and also a ruleset (herein called demo) that uses that ruleset as a module and reacts to events that call for actual work.

Hypothetical API

During this blog post we will make use of a hypothetical API that provides just two calls. These calls are wrapped by the sdk ruleset (whose complete source code is shown in the linked page). They are, in alphabetic order:

/action which, we suppose, actually does something useful (given a valid token)
/token which will supply the token we will need (given a client ID and secret)

The wrapper ruleset provides a defaction (named takeAction) to call the /action API call, and a couple of rules that work with the /token API call. The complete source code for these will be shown in-line as they are discussed.

Requesting a token

In time sequence, the two API calls need to be done in the opposite of alphabetic order, because we need a token before we can take the action.

Here is a rule that will request the client token:

  rule generateNewToken {
    select when sdk new_token_needed
    pre {
      creds = {"username":ClientID,"password":ClientSecret,}
      data = {"grant_type":"client_credentials"}
    }
    http:post(api_url+"/token",auth=creds,form=data) setting(resp)
    fired {
      ent:token :=
        resp{"status_code"}==200 => resp{"content"}.decode()
                                  | null
      ent:issued := time:now()
    }
  }

Notice that this special API uses the client identifier and secret to provide basic authentication, and that we are requesting a client credential (which conforms to one of the OAuth flows).

Hypothetical token

So that the sdk ruleset will compile, we have supposed that the API is located at https://example.com/api which if we were to use it would not complain (since it is example.com which anyone can use) but it would not return a token, so the next paragraph is hypothetical.

Let's suppose that the token returned looks like this (edited for brevity/redaction):

{
  "access_token": "ObI_iQV…3GW-B…Jc.Kl_uz…Gnu095jY",
  "expires_in": 3599
}

So the actual token is one element of the returned value, and we are also told how many seconds it will be valid (just under one hour).

Checking for token validity

Here is a rule that will check if the token is still valid, and if not request a new one:

  rule checkForValidToken {
    select when sdk might_need_a_valid_token
    pre {
      tokenValid = function(now){
        ttl = ent:token{"expires_in"} - 60 // with a minute to spare
        expiredTime = time:add(ent:issued,{"seconds":ttl})
        ent:token{"access_token"} && (expiredTime > now)
      }
    }
    if not tokenValid(time:now()) then noop()
    fired {
      raise sdk event "new_token_needed"
    }
  }

We compare the current time to the time when the token is set to expire. If the token is near expiration or has expired, we raise the sdk:new_token_needed event, and the first rule will be evaluated, which should result in a new token, and give us another hour to work.

Using the token

Having a way to get the token, and to ensure it is valid, we can get to work.

Here we assume that the API has a way to take action, so the sdk ruleset will provide a defaction, named takeAction, that our demo ruleset will be able to invoke to do actual work:

    hdrs = function(){
      {
        "Content-Type":"application/json",
        "Authorization":"Bearer "+ent:token{"access_token"}
      }
    }
    takeAction = defaction(){
      url = api_url + "/action"
      http:put(url,headers=hdrs()) setting(resp)
      return resp
    }

The helper function, hdrs, produces the HTTP headers that we need to use the action API, including the token. These have to be computed in a function so that they include the current value of the access token.

Using the wrapped API

Disclaimer

In this section, we will consider three possible rulesets that attempt to use sdk as a module. The first two, frustratingly, won't actually work. This will give us an opportunity to learn about the input queue or bus that the pico engine provides for each pico that it hosts, and about the schedule of rules to be evaluated. Complete source code for all three rulesets is available in the same folder as the sdk ruleset, with all pertinent code shown here in-line.

First attempt

We are now ready to consider a ruleset that can use the sdk ruleset (as a module) to do some needed work by performing the action.

The pico in which this ruleset is installed wants to react to a demo:work_needed event by calling the sdk's /action API call. As we see, it will call it by invoking the defaction defined in the sdk module (using the action sdk:takeAction in the second rule).

ruleset demo1 {
  meta {
    use module sdk
  }
  rule makeItHappen {
    select when demo work_needed
    fired {
      raise sdk event "might_need_a_valid_token"
    }
  }
  rule reallyMakeItHappen {
    select when demo work_needed
    sdk:takeAction() setting(resp)
    fired {
      raise demo event "real_work_done"
        attributes event:attrs.put("resp",resp)
    }
  }
}

This rule is intended to react to a demo:work_needed event by

ensuring that the sdk has a valid token
taking action using the action API (as wrapped by takeAction)

Why demo1 doesn't work

Sadly, this attempt doesn't work. When the event comes in to the pico (in which we have installed both rulesets), the pico engine queues it up, and as soon as the pico is not busy (when the event comes to the front of the queue), the pico engine determines that the pico needs to evaluate these rules (shown here as ruleset id then rule name):

demo1 makeItHappen
demo1 reallyMakeItHappen

We call this a "schedule" of rules to be evaluated.

After evaluating the first rule, the schedule looks like this, because the first rule to be evaluated raised the sdk:might_need_a_valid_token event:

~~demo1 makeItHappen~~
demo1 reallyMakeItHappen
sdk checkForValidToken

Oh! The next rule to be evaluated is the one named reallyMakeItHappen, and we haven't even begun to ensure that we have a valid token yet!

Trying to fix demo1

Our first attempt to fix this is shown here:

ruleset demo2 {
  meta {
    use module sdk
  }
  rule makeItHappen {
    select when demo work_needed
    fired {
      raise sdk event "might_need_a_valid_token"
      raise demo event "work_needed_using_token"
        attributes event:attrs
    }
  }
  rule reallyMakeItHappen {
    select when demo work_needed_using_token
    sdk:takeAction() setting(resp)
    fired {
      raise demo event "real_work_done"
        attributes event:attrs.put("resp",resp)
    }
  }
}

and it doesn't work either (sometimes it does, and sometimes it doesn't -- the worst kind of bug)! Aargh.

The initial schedule has just one rule:

demo2 makeItHappen

and after the first (and only, in this case) rule is evaluated, the schedule looks like:

~~demo2 makeItHappen~~
sdk checkForValid Token
demo2 reallyMakeItHappen

which looks a lot more promising. The problem comes, in the case where the sdk token has expired, after evaluating the second rule of the schedule, when the schedule becomes:

~~demo2 makeItHappen~~
~~sdk checkForValidToken~~
demo2 reallyMakeItHappen
sdk generateNewToken

Once again, we are going to try to use the takeAction action without having a valid token, and this is the case where we really need a new token.

The correct way to use the wrapped API

We need to have the sdk complete its checking and (possibly) generating a new token before we proceed with our use of the API. That can be done in this way, replacing just the makeItHappen rule:

ruleset demo3 {
  meta {
    use module sdk
  }
  rule makeItHappen {
    select when demo work_needed
    every {
      event:send({"eci":meta:eci,
                  "domain":"sdk","type":"might_need_a_valid_token"})
      event:send({"eci":meta:eci,
                  "domain":"demo","type":"work_needed_using_token",
                  "attrs":event:attrs})
    }
  }
  rule reallyMakeItHappen {
    select when demo work_needed_using_token
    sdk:takeAction() setting(resp)
    fired {
      raise demo event "real_work_done"
        attributes event:attrs.put("resp",resp)
    }
  }
}

Now only one rule gets scheduled when our pico reacts to the demo:work_needed event. It merely adds two new events to the pico's queue, which will be considered in the order they arrive, and only after it has completed its work on the demo:work_needed event.

The schedule for the first sdk:might_need_a_valid_token event, after it has completed, is:

~~sdk checkForValidToken~~
~~sdk generateNewToken~~

The next event to be considered by the pico is demo:work_needed_using_token, and it requires evaluating just the one rule, but since we have just generated a new token, it is sure to be valid.

Notes on demo3

We have to pass through the event attributes to the demo:work_needed_using_token event, but the sdk doesn't need them, so we don't pass them to its event.

Here we are using the same Event Channel Identifier (ECI ) that was used to get the demo:work_needed event to us, meta:eci, to queue up the next two events. This will work only if the policy associated with this channel allows for those events. If not, we would have to locate a channel whose policy does, and use it in place of meta:eci.

Lesson learned

We have learned that the raise event instruction (in the postlude of a rule) causes one or more new rules to be added to the end of the current schedule for rule evaluation. This has important consequences, which hopefully are now clear.

In contrast, an event:send action causes an entirely new event to be added to a pico's queue. It will not be considered at all until later, when the pico has finished the thread it is working on and the new event comes to the head of the queue.

This diagram, taken from Event Loop Introduction, shows the interplay between the pico's queue and the schedule of rules to evaluate. Notice that a pico typically spends most of its time waiting, but occasionally becomes busy for awhile if a number of events have queued up.

For completeness

The code shown here is only for the "happy path" where we always have a valid token when we want to take the action. If for some reason we take the action without a valid token, the response will have a status code of 401 Unauthorized.

More complete code would check for this case after each attempt, and try again after having obtained a new token. How this can be done will be discussed in a subsequent post.

PicoStack

Web API needing a token grant