Speak friend and enter!

Authenticating web requests with yada

Speak friend and enter!

June 2, 2017
Malcolm Sparks

This article is about authentication in a website or web API using our yada library.

We'll start with the basics and build up gradually, covering the following topics:

  • HTTP Basic authentication
  • Custom authentication
  • JSON Web Tokens
  • Login forms

Getting started

If you don't already have a project that uses yada, a good option is to clone our Edge project and fire it up.

git clone https://github.com/juxt/edge
cd edge
boot dev

You'll find Edge contains examples similar to the ones in this tutorial, so you can modify that code and experiment.

Create a restricted resource

Let's begin by creating a function that returns some secret HTML content that greets an authorized user. For speed, we'll use hiccup to generate the HTML:

(require '[hiccup.page :refer [html5]])

(defn- restricted-content [ctx]
      [:h1 (format "Hello %s!"
             (get-in ctx [:authentication "default" :user]))]
      [:p "You're accessing a restricted resource!"]]))

Basic authentication

The first authentication scheme we'll cover is HTTP Basic authentication.

We'll create a yada resource with a GET method that calls this restricted function to generate its response.

Here's a bidi route structure that contains this resource:

    {:id :restricted
     {:get {:produces "text/html"
            :response (fn [ctx] (restricted-content ctx))}}

     {:scheme "Basic"
      :verify (fn [[user password]]
               (when (= user "alice")
                {:user user
                 :roles #{:user}}))
      :authorization {:methods {:get :user}}}})]]]

We've also added an :access-control entry to restrict access to the GET method to users who have the role :user.

Now, insert this bidi route into a web project. If you're using edge, you can find the code in src/edge/examples.clj.

Let's focus on the :access-control section:

{:scheme "Basic"
 :verify (fn [[user password]]
           (when (= user "alice")
             {:user user
              :roles #{:user}}))
 :authorization {:methods {:get :user}}}

The :scheme entry is set to "Basic". In this scheme, yada looks for an Authorization header in the request with a value beginning with Basic. If this exists, the user and password are decoded and passed to a function that you provide, which is our next entry, :verify.

The :verify function should ensure that the user and password given are correct, and if so, return a map of credentials. You can return anything you like in this map, but :roles is a bit special, because it's checked by the default authorization scheme used by yada (which you can, of course, override with your own).

The authentication process happens regardless of whether the page is public or restricted. To restrict access, we must also specify an :authorization entry:

:authorization {:methods {:get :user}}

These specifies that all GET requests must establish the :user role in order for the request to proceed.


Enough with the theory, let's test things are working as expected,

Use curl (or a browser, hitting cancel if presented with the authentication dialog) to hit this restricted resource. For example, if you're using edge, from a terminal do this:

$ curl -i http://localhost:3000/authn-examples/basic

You should get a 401 status code (look for it in the first line), meaning the request is unauthorized:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="default"
Content-Length: 102
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Mon, 13 Feb 2017 18:54:43 GMT

No authorization provided

Also, note the presence of the WWW-Authenticate header, prompting you to provide credentials for the Basic authentication scheme, for the domain "default".

Now repeat the request but add the username of alice with a password (which can be anything you like) to the curl request:

$ curl -i http://localhost:3000/authn-examples/basic -u alice:foo

This should get you access to the restricted resource:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 139
Content-Type: text/html
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Fri, 07 Apr 2017 23:24:35 GMT

<body><h1>Hello alice!</h1><p>You're accessing a restricted resource!</p></body>


You may have wondered why a 401 error means 'Unauthorized' rather than 'Unauthenticated'.

The lack of authentication, alone, is not sufficient to deny access to a page. Your browser probably isn't sending authentication credentials to this blog article – yet you're still authorized to read it!

It's the authorization requirement that causes the 401. Try commenting out the :authorization entry in the example and trying again, you'll get a 200.

As a brief digression, put back the :authorization entry but change the :roles returned from the :verify function to exclude the :user role. As long as you still authenticate correctly you should now get a 403 Forbidden, indicating that the request has been authenticated and yet still has insufficient rights to access the resource.

These semantics are defined by the HTTP specification, but most web libraries define their own (so don't worry if you're having trouble grokking this stuff, you're not alone!)

Custom authorization in yada is a separate topic which we'll cover in a future blog post, so let's keep the focus on authentication.

Custom authentication

Let's add another resource, by copying the code of first restricted resource, but changing the :access-control entry to the following:

{:scheme :my/custom-static
 :authorization {:methods {:get :user}}}

Note the :scheme has changed from "Basic" to :my/custom-static and notice there's no :verify function this time (we won't need one).

If we send a request to this URL we'll get the same 401 response:

HTTP/1.1 401 Unauthorized
Content-Length: 56
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Sun, 12 Feb 2017 14:01:47 GMT

No authorization provided

This time, there's no WWW-Authenticate header. You should also notice a warning present in the server's log.

WARN yada.security - No installed support for the following scheme: :my/custom-static

What's going on here? By declaring a new authentication scheme, yada's authentication code attempts to call the yada.security/verify multimethod, which dispatches on the declared authentication scheme, in our case :my/custom-static, for which there is no defmethod registered yet. So let's add it!

We'll provide a naïve implementation that returns the user and roles as a constant (without even accessing the request):

(defmethod yada.security/verify :my/custom-static
  [ctx scheme]
  {:user "alice"
   :roles #{:user}})

Let's see what happens:

$ curl -i http://localhost:3000/authn-examples/custom-static

We should get through the security and see our restricted resource returned with a 200 status code.

HTTP/1.1 200 OK

Request-based authentication

Right now our authentication method is going to let every request through..

Let's add a new defmethod for a different scheme, this time we'll call it :my/trusted-whoami-header:

(defmethod verify :my/trusted-whoami-header
  [ctx scheme]
  (when-let [user (get-in ctx [:request :headers "x-whoami"])]
    {:user user
     :roles #{:user}}))

This time, we're pulling out the X-WhoAmI header from the request. If it's there, we set the user to the value of the header.

Since header names are case-insensitive, all header names are converted to lower-case to avoid misses.

Let's test it.

$ curl -i http://localhost:3000/authn-examples/custom-trusted-header

You should get a 401.

For the example above, I've used bidi to bind this modified resource to the path /authn-examples/custom-trusted-header (this keeps things aligned with the examples in Edge). So if you're getting a 400, check your bidi route structure and the request's URL.

Now let's add the header:

$ curl -i http://localhost:3000/authn-examples/custom-trusted-header -H X-WhoAmI:bob

You should get a 200.

Adding trust with digital signatures

Our resource now does authentication, but is not secure. Unauthorized users can easily forge the X-WhoAmI header. How can we add some trust to this header?

One solution is for us to sign the header value with a digital signature based on some secret. If the credentials are tampered with, we'll detect this when we verify the signature.

Since we have a way of trusting the information we're signing in the request, we can issue and sign the entire map of credentials in one go. This saves us from having to make trips to the database to look up users and roles on each request, with the trade-off that we won't be able to determine if these credentials have been revoked.

We can try this now by using the buddy-sign library. This provides an implementation of JSON Web Token (JWT) which is a popular approach to digital signatures on the web. Buddy is built into yada by default, so should be in your classpath already.

Let's go to our REPL and require it:

user> (require '[buddy.sign.jwt :as jwt])

Now let's create a secret which we'll use when signing and verifying signatures:

user> (def secret "ieXai7aiWafeSh6oowow")

Now let's sign some credentials!

Here's one for Alice:

user> (jwt/sign {:claims (pr-str {:user "alice" :roles #{:user}})} secret)

And here's one, with an empty set of roles, for Dave:

user> (jwt/sign {:claims (pr-str {:user "dave" :roles #{}})} secret)

(By the way, the reason we use pr-str is to write out our Clojure credentials map as a string. JWT doesn't support Clojure's keywords and sets, but we can easily read back this string later with clojure.edn).

Our verify function now needs to change to check the signature and read our claims:

(require '[clojure.edn :as edn])

(defmethod verify :my/signed-whoami-header
  [ctx scheme]
   (get-in ctx [:request :headers "x-whoami"])
   (jwt/unsign "ieXai7aiWafeSh6oowow")

Create a new resource (with copy-and-paste!) that uses this new authentication scheme. Add a bidi route, or equivalent, to bind the resource to /authn-examples/custom-signed-header.

Now let's test we can still get access:

$ curl -i http://localhost:3000/authn-examples/custom-signed-header -H x-whoami:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGFpbXMiOiJ7OnVzZXIgXCJhbGljZVwiLCA6cm9sZXMgI3s6dXNlcn19In0.yMoqLM3zPkej-W6CoEaJ7GWxxrsbEiYa_yiRw7rPDmU

If you are having difficulty understanding what is going on here, try seeing what happens when you try Dave's JWT:

$ curl -i http://localhost:3000/authn-examples/custom-signed-header -H x-whoami:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGFpbXMiOiJ7OnVzZXIgXCJkYXZlXCIsIDpyb2xlcyAje319In0.Kd90HYTQFDC-ZAC95CvrnY1PWQXGBkLMOzS2lnC9ZA8

Since Dave doesn't have any roles, and since the :user role is required by the :authorization scheme for the resource, Dave gets back a 403:

HTTP/1.1 403 Forbidden
Content-Length: 40
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Tue, 30 May 2017 08:10:50 GMT


As an exercise, what happens when you tamper with the JWT?

Further considerations with JWT

In our current implementation of verify we let any exceptions from Buddy pass through, like those that are thrown if the signature is invalid. These ultimately yield a 500 error. But we could decide to return nil (causing a 401) or even alert an internal security system that someone is trying to crack our security.

Note that one significant downside to JWT is that we can't revoke a signature. That means the JWT is valid until we change the secret on which the signature is based. If we've used the same signature for signing other JWTs, we'd have to revoke everybody's JWTs or none at all.

There are a number of solutions open to us. One is to add an expiry date to the claims, or hold a blacklist in memory that is notified of any changes.

Form-based logins

The problem with our current design is that we now have the problem of distributing these signed tokens to end-users. How can we be certain they won't get intercepted by a malicious party?

We can solve this by asking the user to prove their identity via a form, in exchange for a cookie containing the signed token. That way, all subsequent requests will send back the cookie and we'll be able to verify the credentials on each request.

This is such a common pattern that we see it all the time: the humble website login dialog.

Let's create a new yada resource:

    {:get {}
     :post {}}})]

Here we have a yada resource with 2 methods, GET and POST.

Add this code as the :get method entry:

{:produces "text/html"
 (fn [ctx]
    [:form {:method :post}
      [:label {:for "user"} "User "]
      [:input {:type :text :id "user" :name "user"}]]
      [:label {:for "password"} "Password "]
      [:input {:type :password :id "password" :name "password"}]]
      [:input {:type :submit}]]]))}

This produces a simple HTML form, which submits its fields via a POST method to the same resource.

Here's the :post entry:

{:consumes "application/x-www-form-urlencoded"
 :produces "text/plain"
 {:form {:user String
         :password s/Str}}
 (fn [ctx]
   (let [{:keys [user password]} (-> ctx :parameters :form)]
      (:response ctx)
      (if-not (#{"alice" "dave"} user)
        {:body "Login failed"
         :status 401}
        {:status 303
         :headers {"location" (yada/url-for ctx :restricted)}
            {:claims (pr-str {:user user :roles #{:user}})}

The POST method declares it consumes an encoded HTML form in the body of the request, which contains user and password parameters (yada will return a 400 response for requests that don't).

Now onto the response function. If the user is neither alice nor dave, we fail the login with a 401 and a message. Otherwise, we treat the login as successful.

Normally we'd be checking password hashes here too, but that's outside of the scope of this article. (If you're going to do this, don't forget to hash your passwords with something like bcrypt - again, buddy can help.)

If the login is successful we set a cookie called session (it can be called anything you like), and we set the value to our JWT token as before, this time using the user parameter entered in the form.

By the way, in yada, we return an explicit Ring response by starting with the :response entry in the request context, and augmenting it as necessary. This lets yada know we want to specify the Ring response ourselves — if we just return a normal map, yada considers that's some data we want to return, rather than an explicit Ring response.

We're not quite done yet. On a successful login we want to redirect away from the login page, to a resource that will show the restricted content. The URL for this can be acquired from bidi, indirectly via yada's url-for function: (yada/url-for ctx :restricted). We use :restricted to reference a resource with that :id. Here it is:

(def cookie-based-restricted-resource
   {:id :restricted
    {:get {:produces "text/html"
           :response (fn [ctx] (restricted-content ctx))}}
    {:scheme :my/signed-cookie
     :authorization {:methods {:get :user}}}}))

Note the authentication scheme needs to extract the cookie and verify the signature. Here's the adjusted defmethod we'll need:

(defmethod verify :my/signed-cookie
  [ctx scheme]
   (get-in ctx [:cookies "session"])
   (jwt/unsign secret)

Time to test our form. Use a browser this time and enter the URL for your /authn-examples/login resource. Login with alice and check that you can now access the restricted the resource. Try again with another user, chuck, to ensure you get the 401.


This has been a long article, but authentication of web requests is an important and lengthy topic.

Rather than provide fixed solutions to authentication, yada provides the pieces for you to compose your own. We'll demonstrate this in a future article where we assemble some yada components to provide OAuth2 authentication.

This is very much in the spirit of Clojure: libraries provide the productivity boost of proven pre-written code, without forcing you to sacrifice the overall control you need over your project. This is a rare combination in software development, yet I believe a strong factor in the success of software projects.

That's it for now. I hope you found the article useful. Do feel free to raise any questions on our yada support channels, on Slack and our yada-discuss mailing list.