Preface
State is everywhere. The world is moving and we need to keep up. We need our computers to help us stay up-to-date with the latest information, chats, trends, stock-prices, news and weather updates, and other important stuff.
The web is primarily a means to move state around. You have some state here, and you want it over there. Or it’s over there, but you want it over here.
For two decades or more, the pre-dominant model for web programming has ignored state, instead requiring developers to work at the level of the HTTP protocol itself.
For example, in Java:
public void handleRequest(HttpServletRequest request,
HttpServletResponse response)
{
response.setStatus(200);
}
or in Clojure
(fn [request] {:status 200})
This programming model puts the HTTP request and response at centre stage. The concept of state is missing entirely - the resource is seen merely as an operation (or set of operations) available for remote invocation.
For years, the same RPC-centered approach has been copied by web frameworks in many languages, old and new (Python, Ruby, Go, Clojure, Rust…). It has survived because it’s so flexible, as many low-level programming models are.
But there are significant downsides to this model too. HTTP is a big specification, and it’s unreasonable to expect developers to have the time to implement all the relevant pieces of it. What’s more, many developers tend to implement much the same code over and over again, for each and every operation they write.
A notable departure from this programming model can be found in Erlang’s WebMachine and Clojure’s Liberator. To a degree, these libraries ease the burden on the developer by orchestrating the steps required to build a response to a web request. However, developers are still required to understand the state transition diagram underlying this orchestration if they are to successfully exploit these libraries to the maximum extent. Fundamentally, the programming model is the same: the developer is still writing code with a view to forming a response at the protocol level.
While this model has served us well in the past, there are increasingly important reasons why we need an upgrade. Rather than mere playthings, HTTP-based APIs are becoming critical components in virtually every organisation. With supporting infrastructure such as proxies, API gateways and monitoring, there has never been a greater need to improve compatibility through better conformance with HTTP standards. Yet many APIs today at best ignore, and at worst violate many parts of the HTTP standard. For ephemeral prototypes, this casual approach to HTTP is acceptable. Yet HTTP is designed for long-lived systems with lifetimes measured in decades, that must cross departmental and organisational boundaries, while adapting to ongoing changes in technology.
It’s time for a fresh approach. We need our libraries to do more work for us. For this to happen, we need to move from the de-facto operational view of web services to a strong data-oriented approach, focussing on what a web resource is really about: state.
Basics
1. Introduction
yada is a web library for Clojure that lets you define websites and web APIs using data.
Despite some differences, virtually all web libraries do a similar thing: call your code with some request data and ask you to return a response, leaving the bulk of the responsibililty for implementing the HTTP standards in your hands: responding with the correct status codes, response headers and ensuring semantics have been followed properly.
yada is different. Rather than leaving all the implementation details to you, it helps you out in doing the right thing (according to HTTP standards), thereby creating a better web [1].
It achieves this by providing you with a highly-configurable handler that is general enough to use in the vast majority of cases. Rather than coding the handler, the developer only has to configure one. This declarative approach provides greater scope for accuracy, consistency and re-use.
yada represents a break from the traditional yet stale method of building web backends, an approach that can trace its roots all the way back to the days of Common Gateway Interface. While yada's data-oriented approach takes full advantage of the data-oriented philosophy of Clojure, there’s no reason why this methodology cannot be implemented by new web libraries in other languages.
Using yada for your web project will allow you to finally exploit and benefit from the many features embodied in the outstanding design of HTTP, for scale, flexibility, security, longevity and interoperability.
1.1. Design goals
yada is designed to meet the following goals:
-
Be easy to use for intermediate Clojure developers
-
Comprehensive compliance with HTTP standards over pragmatism and performance
-
Increase productivity through re-use
-
Handle large workloads with reasonable performance
-
Support multiple architectural styles, including Hypermedia APIs (REST)
yada is not an experiment. It is designed for, and has been tested in, production environments.
yada is sufficiently quick-and-easy for quick prototype work but scales up when you need it to, to feature-rich secure services that can handle the most demanding workloads, while remaining faithful to the HTTP standards.
Some familiarity with HTTP will help you understand yada concepts quicker, but isn’t absolutely necessary. As you learn how to wield yada you will also discover and learn more about the HTTP standards as you go.
1.2. Say Hello! to yada
It’s quick to get started with yada without knowing how it works - it’s easy to get started even if you only have a basic knowledge of Clojure.
Let’s begin with a few examples. The obligatory Hello World! example is (yada/handler "Hello World!")
, which responds with a message.
Perhaps you might want to serve a file? That’s
(yada/handler (new java.io.File "index.html"))
.
Now you know how to serve a file, you know how to serve a directory. But perhaps you’ve got some resources on the classpath?
(yada/handler (clojure.java.io/resource
"talks/"))
.
What about (yada/handler nil)
? Without knowing, can you guess what that might do? (That’s right, it produces a 404 Not Found
response).
What about a quick dice generator? (yada/handler #(inc (rand-int 6)))
.
Notice we use a function here, rather than a constant value, to vary numbers between rolls.
How about streaming those dice rolls as Server Sent Events? Put those dice rolls on a core.async channel, and return it with yada.
All these examples demonstrate the use of Clojure types that are converted on-the-fly into yada resources, and you can create your own types too.
Let’s delve a little deeper…
1.3. Resources
In yada, resources are defined by a plain-old Clojure map.
This has many benefits. While functions are opaque, data is open to inspection. Data structures are easy to generate, transform and query - chores that Clojure makes light work of.
Here’s an example of a resource:
(yada/resource
{:properties {…}
:methods {:get {:response (fn [ctx] "Hello World!")}
:put {…}
:brew {…}}
…
})
There’s a lot of things you can do with a resource data model but perhaps the most obvious is to create a request handler from it to create responses from HTTP requests. That’s the role of a handler.
With yada, we transform a resource into a handler using the handler
function.
(require '[yada.yada :as yada])
(yada/handler (yada/resource {…}))
A handler can be called as a function, with a single argument representing an HTTP request. It returns a value representing the corresponding HTTP response.
A yada handler is an instance of the yada.handler/Handler record. Since this record satisfies clojure.lang.IFn , yada handlers behave just like normal Ring handlers and can be used wherever you might use a Ring handler.
|
1.4. Serving requests
To use yada to create real responses to real HTTP requests, you need to add yada to a web-server. The web server takes care of the networking and messages of HTTP (RFC 7230), while yada focuses on the semantics and content ([RFC7231] and upwards).
Currently, the only web server you can use is ztellman/aleph. This is because yada is built on an asynchronous abstraction provided by ztellman/manifold. However, there is no technical reason why, in future, other web servers can’t be wrapped by manifold. In the meantime, Aleph (which wraps a much bigger library, Netty) provides a very capable and performant server. |
To write real applications you also need a router that understands URIs, and yada has some features that are enabled when used with juxt/bidi, although there is nothing to stop you using yada with other routing libraries.
1.5. Conclusion
That’s yada in a nutshell, but the quickest way to to learn it is to set up an environment and play. That’s what we’ll do in the next chapter.
2. Getting Started
In this quick tutorial we’re going to run a real Clojure project, diving into the code to show how yada is used.
Our project is called Edge, a sample project from JUXT to show some of our libraries in action. It lives on GitHub.
We’ll clone it first, then build it, then run it, then browse the examples and even make modifications.
So let’s get going!
2.1. Clone
First let’s clone the project and change into its working directory.
git clone https://github.com/juxt/edge cd edge/app
2.2. Build & Run
Next we build and run it, in development mode.
clojure -A:dev:build:dev/rebel
This can take up to a couple of minutes to build and run from scratch so don’t worry if you have to wait a bit before you see anything.
[Edge] Starting nREPL server [Edge] nREPL client can be connected to port 5600 [Rebel readline] Type :repl/help for online help info [Edge] Loading Clojure code, please wait... Figwheel: Starting server at http://0.0.0.0:3449 Figwheel: Watching build - main Compiling "target/public/edge.js" from ("/home/username/Projects/clj/edge/app/dev" "/home/username/Projects/clj/edge/app/test" "/home/username/Projects/clj/edge/app/aliases/rebel" "/home/username/Projects/clj/edge/app/src" "/home/username/Projects/clj/edge/app/sass" "/home/username/Projects/clj/edge/app/resources" "/home/username/Projects/clj/edge/app/assets" "/home/username/.gitlibs/libs/io.dominic/krei.alpha/02d0675365d76e81cd2392e7f397e6f278e2a118/src")... Successfully compiled "target/public/edge.js" in 3.472 seconds. Figwheel: Starting CSS Watcher for paths ["target"] [Edge] Now enter (go) to start the dev system dev=>
2.3. Start the Server
At the dev⇒ prompt enter (go) to start a server listening by default on port 3000.
dev=> (go) [Edge] Website can be browsed at http://localhost:3000/ [Edge] Now make code changes, then enter (reset) here :started
2.4. Browse
Fire up a browser and browse to http://localhost:3000/hello. You should see a simple Hello World
message.
2.5. Working with the REPL
We’re going to start changing some of Edge’s source code soon, and when we do that we’ll type (reset)
on our REPL. So let’s try that now.
dev=> (reset) :reloading (io.dominic.krei.alpha.impl.util io.dominic.krei.alpha.core edge.phonebook.db edge.lacinia edge.phonebook-app edge.selmer edge.test.system edge.phonebook edge.sources edge.hello edge.examples edge.web-server edge.system edge.examples-test edge.system-test edge.main dev user io.dominic.krei.alpha.main edge.rebel.main edge.api-test) :resumed dev=>
2.6. Test the service
Let’s send an HTTP request to the system to check it is working. We can use a browser to visit http://localhost:3000/hello or use curl
if you have it installed on your system:
curl http://localhost:3000/hello
The result should be the same:
Hello World!
2.7. Locate the source code
Fire up an editor and load up the file src/edge/hello.clj
.
Locate the function called hello-routes
. This returns a simple route structure that matches on the URI paths of incoming HTTP requests.
(defn hello-routes [_]
["/hello" (yada/handler "Hello World!\n")])
Make a change to string "Hello World!"
, for example, change it to "Hello Wonderful World!"
.
2.8. Reset the system
Now we’ve made a change to Edge’s source code, we must tell the system to reset. The system will then detect all the code changes and necessary dependencies to reload.
dev=> (reset) :reloading (io.dominic.krei.alpha.impl.util io.dominic.krei.alpha.core edge.phonebook.db edge.lacinia edge.phonebook-app edge.selmer edge.test.system edge.phonebook edge.sources edge.hello edge.examples edge.web-server edge.system edge.examples-test edge.system-test edge.main dev user io.dominic.krei.alpha.main edge.rebel.main edge.api-test) :resumed dev=>
Let’s test the service again:
$ curl http://localhost:3000/hello
You should now see that the change has been made:
Hello Wonderful World!
Congratulations. You’re all up and running with a project built with yada. This will make a great lab to try out your own yada experiments and see what is possible.
Browse to http://localhost:3000 and have fun!
3. Hello World!
In this chapter we examine the "Hello World!" resource in depth. If you have followed the Getting Started chapter, you’ll be up and running and ready to try the examples yourself, but it’s up to you.
Let’s look at the route definition in the file src/edge/hello.clj
(the same one we saw in the Getting Started chapter). This function returns a route structure. The simplest route structure is a pair of elements contained in a vector.
(defn hello-routes [deps]
["/hello" (1)
(yada/handler "Hello World!\n") (2)
])
1 | The first element is a path (or pattern) which matches on the URI’s path when handling HTTP requests. |
2 | The second element is often a handler function (but can be another route structure, recursively). |
In this route structure, any incoming request with a path of /hello
gets sent to the yada handler defined by (yada/handler "Hello World!\n")
.
Let’s focus on this handler. We could have used a standard Ring function (fn [req] {…})
here but instead we created one with yada/handler
, which takes a resource and turns it into a handler function.
A resource is a Clojure map (or more accurately, a Clojure record) that completely describes the methods, properties, representations, security and other miscellaneous properties of a web resource, as data. The reason we can use a string ("Hello World"
) here is because yada contains logic to coerce a string into a resource.
There are a number of built-in coercions from various Clojure types and of course you can provide your own. |
3.1. Examining the response
Let’s send a request which gets routed to our handler, which creates the response. Let’s examine this response in more detail, via curl.
$ curl -i http://localhost:3000/hello
The -i option to curl shows us the HTTP status and response headers as well as the body, which is very useful for debugging. If you ever need to see the request headers too, add -v .
|
The response should be something like following (note that headers may appear in a different order):
HTTP/1.1 200 OK Server: Aleph/0.4.1 Connection: Keep-Alive Date: Fri, 17 Jun 2016 16:44:23 GMT Last-Modified: Fri, 17 Jun 2016 16:43:02 GMT ETag: fa863bd7ff53786d286e4bb3c0134416 Content-Type: text/plain;charset=utf-8 Vary: accept-charset Content-Length: 23 X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff
The first three response headers are added by our webserver, Aleph.
Server: Aleph/0.4.0 Connection: Keep-Alive Date: Fri, 17 Jun 2016 16:44:23 GMT
Next we have another date and a string known as the entity tag:
Last-Modified: Sun, 09 Aug 2015 07:25:10 GMT ETag: fa863bd7ff53786d286e4bb3c0134416
The Last-Modified
header shows when the string Hello World!
was created, which happens to be the last time the system was started (or reset). Java strings are immutable, so yada is able to deduce that the
string’s creation date is also the last time it could have been modified.
The entity tag is computed from the value of the Hello World!
itself. Unlike the Last-Modified
value, it can survive a reset.
Both Last-Modified
and ETag
are used to support HTTP conditional requests and conflict detection when uploading new versions of a resource.
Next we have a header telling us the media-type of the string’s representation.
Content-Type: text/plain;charset=utf-8
yada is able to determine that the media-type is text, but without more information it must default to text/plain
. It can also tell us the charset, which defaults to the default charset of the JVM (almost always utf-8
)
Vary: accept-charset
Since the Java platform can encode a string to a number of charsets, yada adds a Vary header to signal to the user-agent (and proxy caches in between) that the body can change if a request contained a different Accept-Charset header. Java installations support many different charsets, so yada does too.
Next we are given the length of the body.
Content-Length: 13
This value is in bytes, regardless of the charset. It includes the newline.
Finally we see our response body.
Hello World!
3.2. A conditional request
In HTTP, a conditional request is one where a user-agent (like a browser) can ask a server for the state of the resource but only if a particular condition holds.
A common condition is whether the resource has been modified since a particular date, usually because the user-agent already has a copy of the resource’s state which it can use if possible. If the resource hasn’t been modified since this date, the server can tell the user-agent that there is no new version of the state.
We can test this by setting the If-Modified-Since header in the request.
Here’s how we might do this using the curl command.
$ curl -i http://localhost:3000/hello -H "If-Modified-Since: Mon, 1 Jan 2525 00:00:00 GMT"
Of course, nobody will have modified the resource since the year 2525, so we should get a 304 response, telling us we can use our cached copy:
HTTP/1.1 304 Not Modified X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 0 Vary: accept-charset ETag: fa863bd7ff53786d286e4bb3c0134416 Server: Aleph/0.4.1 Connection: Keep-Alive Date: Wed, 22 Jun 2016 16:57:46 GMT
Notice we also get the same Vary
and ETag
headers. These help any proxies between the user-agent and the service properly cache content, and if they would have been produced in a 200 response, then they must also be produced in a 304. (This is the kind of thing that yada takes care of for you, unlike most libraries).
3.3. Content negotiation
The responses we have received back from our service all contain this curious header called Vary
set to accept-charset
. The server is telling us (and any proxies between us) that the representation might vary depending on the charset negotiated. Let’s see if we can get our "Hello World!" message returned in other charsets.
Let’s try getting the string in UTF16 by telling the server that’s the only charset we’ll accept:
curl -i http://localhost:3000/hello -H "Accept-Charset: UTF-16"
This returns the following:
HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 28 Content-Type: text/plain;charset=utf-16 Last-Modified: Sun, 26 Jun 2016 11:11:31 GMT Vary: accept-charset ETag: 43b1f79e8efe0fa97c32901fbd5746d6 Server: Aleph/0.4.1 Connection: Keep-Alive Date: Mon, 27 Jun 2016 07:45:07 GMT ��Hello World!
The "Hello World!" message is prepended with 2 bytes called the Byte Order Mark (BOM). The length of the string (including the newline) is 13 characters. Since each character here is 2 bytes, that makes 26. The additional of the BOM makes it 28, which is what our Content-Length
header reports.
A BOM indicates the order that the 2 bytes are transmitted in. In big endian form the most-significant byte is transmitted first. We can tell the service that we only want the big endian form with the following:
curl -i http://localhost:3000/hello -H "Accept-Charset: UTF-16BE"
This will now produce the message without the BOM, because it is unnecessary. This means our Content-Length
will be exactly 13 * 2 = 26.
HTTP/1.1 200 OK … Content-Length: 26 Hello World!
If we were to use UTF-32, which defaults to big-endian, we’ll get a Content-Length of 13 * 4 = 52.
HTTP/1.1 200 OK … Content-Length: 52 Hello World!
Note also that different representations generate different ETag
values. The entity tag is a way of managing a cache of representations, not a cache of resources. Think of the ETag
value as the key you could use in a key/value store that stored a cache of representations.
The negotiation of charsets may be considered by some to be unnecessary given the dominance of UTF-8. That is certainly true for today’s modern browsers. However, there are many other types of devices that are being connected to the internet (under the banner Internet of Things). Many of these devices have very tight constraints on processing and memory which prevent them from supporting UTF-8. If we are building a web service, we may want to connect these devices to it in the future.
3.3.1. Languages
Of course it is not just charsets that can be negotiated. Another example is languages. Our "Hello World!" string is in English. Let’s provide support for simplified Chinese.
This calls for a different implementation:
(defn hello-language []
["/hello-language"
(yada/resource (1)
{:methods
{:get (2)
{:produces
{:media-type "text/plain"
:language #{"en" "zh-ch;q=0.9"}} (3)
:response
#(case (yada/language %) (4)
"zh-ch" "你好世界\n"
"en" "Hello World!\n")}}})])
1 | Using the yada/resource function to create a custom resource |
2 | The resource has a single method, GET |
3 | English is preferred, but Simplified Chinese is available too |
4 | This is a function that is given a context as the first argument. The yada/language convenience function pulls out the negotiated language from this context |
Let’s test this by providing a request header which indicates a preference for simplified Chinese:
$ curl -i http://localhost:3000/hello-language -H "Accept-Language: zh-CH"
We should get the following response:
HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 13 Content-Type: text/plain Content-Language: zh-ch Vary: accept-language Server: Aleph/0.4.1 Connection: Keep-Alive Date: Mon, 27 Jun 2016 08:20:59 GMT 你好世界
There’s a lot more to content negotiation than this simple example can show. It is covered in depth in subsequent chapters.
3.4. Mutation
Let’s try to overwrite the string by using a PUT
.
$ curl -i http://localhost:3000/hello -X PUT -d "Hello Wonderful World!%0a"
The response is as follows:
HTTP/1.1 405 Method Not Allowed Allow: GET, HEAD, OPTIONS Content-Length: 284 Content-Type: application/json Server: Aleph/0.4.1 Connection: Keep-Alive Date: Mon, 27 Jun 2016 08:56:58 GMT
The response status is 405 Method Not Allowed
, telling us that our request was unacceptable. There is also an Allow header, telling us which methods are allowed. One of these methods is OPTIONS, which we could have used to check whether PUT was available without actually attempting it.
$ curl -i http://localhost:3000/hello -X OPTIONS
The response should be:
HTTP/1.1 200 OK Allow: GET, HEAD, OPTIONS Content-Length: 0 X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Server: Aleph/0.4.1 Connection: Keep-Alive Date: Mon, 27 Jun 2016 09:00:27 GMT
Both the PUT
and the OPTIONS
response contain an Allow header which tells us that PUT
isn’t possible. This makes sense, because we can’t mutate a Java string.
We could, however, wrap the Java string in a Clojure atom which could reference different Java strings at different times.
To demonstrate this, try the following with the identifier http://localhost:3000/hello-atom
.
(yada/as-resource (atom "Hello World!\n"))
Let’s try a normal GET.
$ curl -i http://localhost:3000/hello-atom
We can now make another OPTIONS
request to see whether PUT
is available, before trying it.
$ curl -i http://localhost:3000/hello-atom -X OPTIONS
HTTP/1.1 200 OK Allow: GET, DELETE, HEAD, OPTIONS, PUT Content-Length: 0 X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Server: Aleph/0.4.1 Connection: Keep-Alive Date: Tue, 05 Jul 2016 15:41:36 GMT
It is! So let’s try it.
$ curl -i http://localhost:3000/hello-atom -X PUT -d "value=Hello Wonderful World!%0a"
And now let’s see if we’ve managed to change the state of the resource.
$ curl -i http://localhost:3000/hello-atom
HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 23 Content-Type: text/plain;charset=utf-8 Last-Modified: Tue, 05 Jul 2016 16:08:22 GMT Vary: accept-charset ETag: 3c3e0684be182b7185f6ad10b63f246a Server: Aleph/0.4.1 Connection: Keep-Alive Date: Tue, 05 Jul 2016 16:08:35 GMT Hello Wonderful World!
As long as someone else hasn’t sneaked in a different state between your PUT
and subsequent GET
, you should see the new state of the resource is "Hello Wonderful World!". Great!
But what if someone did manage to PUT
their change ahead of yours? Their version would now be overwritten. That might not be what you wanted. To ensure we don’t override someone’s change, we could have set the If-Match header using the ETag value.
Let’s test this now, using the ETag value we got before we sent our PUT
request.
$ curl -i http://localhost:3000/hello-atom -X PUT -H "If-Match: fa863bd7ff53786d286e4bb3c0134416" -d "value=Hello Wonderful World!%0a"
HTTP/1.1 412 Precondition Failed Content-Length: 196 Content-Type: application/json Server: Aleph/0.4.1 Connection: Keep-Alive Date: Tue, 05 Jul 2016 16:10:53 GMT
We get a 412, which means a pre-condition failed. The pre-condition in question relates to our If-Match
header value not matching the current value of the atom. This is a very useful result, because it means we can ensure that we don’t overwrite someone else’s data.
3.5. A HEAD request
There was one more method indicated by the Allow header of our OPTIONS
request, which was HEAD
. Let’s try this now.
Use the option --head to curl to tell it to issue a HEAD request (and not to expect a request body). |
$ curl -i --head http://localhost:3000/hello
The response does not have a body, but tells us the headers we would get if we were to try a GET
request.
3.6. Parameters
Often, a resource’s state or behavior will depend on parameters in the request. Let’s say we want to pass a parameter to the resource, via a query parameter.
To show this, we’ll write some real code:
(require '[yada.yada :refer [yada resource]])
(defn say-hello [ctx]
(str "Hello " (get-in ctx [:parameters :query :p]) "!\n"))
(def hello-parameters-resource
(resource
{:methods
{:get
{:parameters {:query {:p String}}
:produces "text/plain"
:response say-hello}}}))
(def handler (yada/handler hello-parameters-resource))
This declares a resource with a GET method, which responds with a plain-text message formed from the query parameter.
Let’s see this in action, but without a parameter:
$ curl -i http://localhost:3000/hello-parameter
Here we get a 400 response. This means we’ve done something wrong (we’ve forgotten to add the query parameter).
Now let’s add the query parameter to the URI:
$ curl -i http://localhost:3000/hello-parameter?p=Ken
This should now get the 200 response we wanted:
HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 11 Content-Type: text/plain Server: Aleph/0.4.1 Connection: Keep-Alive Date: Tue, 05 Jul 2016 16:23:26 GMT Hello Ken!
Great!
As well as query parameters, yada supports path parameters, request headers, form data, cookies and request bodies. You can have optional parameters, in fact, anything that can be expressed in Plumatic Schema, and yada will even coerce parameters to a range of types. For more details, see the parameters chapter.
3.7. Hello Swagger!
Now we have seen how to build a single web resource, let’s see how to build a Swagger description from a collection of web resources.
In your editor, switch to src/edge/web_server.clj
. This file defines the overall route structure which includes our routes for "Hello World!". This has been included twice, both at the root and under the /api
path.
This second version uses the Clojure threading macro ->
which wraps the route structure with yada/swaggered
and gives it a bidi tag (used for generating URIs, we’ll use this later).
[
;; Hello World!
(hello-routes {})
["/api" (-> (hello-routes {})
;; Wrap this route structure in a Swagger
;; wrapper. This introspects the data model and
;; provides a swagger.json file, used by Swagger UI
;; and other tools.
(yada/swaggered
{:info {:title "Edge API"
:version "1.0"
:description "An example API"}
:basePath "/api"})
;; Tag it so we can create an href to this API
(tag :edge.resources/api))]]
The purpose of yada/swaggered
is to augment the route structure given to it with a route to swagger.json
, which responds with a Swagger description of the route structure in JSON. Since yada resources are data maps, this is a relatively simple data transformation of the route structure.
We can test the resource is available at its /api
location with curl:
$ curl -i http://localhost:3000/api/hello
We can also query the Swagger description with curl:
$ curl -i http://localhost:3000/api/swagger.json
This time we get a JSON body returned:
HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 290 Content-Type: application/json Last-Modified: Wed, 22 Jun 2016 15:45:16 GMT Vary: accept-charset ETag: 7833a69510d2b80f2a414c3c4ef2b4d4 Server: Aleph/0.4.1 Connection: Keep-Alive Date: Wed, 22 Jun 2016 15:56:25 GMT {"swagger":"2.0","info":{"title":"Edge API","version":"1.0","description":"An example API"},"produces":["application/json"],"consumes":["application/json"],"paths":{"/hello":{"get":{"produces":["text/plain"],"responses":{"default":{"description":""}}}}},"basePath":"/api","definitions":{}}
Notice we still get a Vary
header telling us that multiple charsets are available. JSON bodies are available in UTF-16 and UTF-32. Compare this with Clojure’s EDN, which is specificed to be UTF-8 only. In fact, yada is happy to produce Swagger definitions in EDN too:
$ curl -i http://localhost:3000/api/swagger.edn
Note that this time we get no Vary
header, since there are no charset alternatives.
HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 284 Content-Type: application/edn Last-Modified: Wed, 22 Jun 2016 15:45:16 GMT ETag: 3aa57341aa88d68108dbead14f5b462c Server: Aleph/0.4.1 Connection: Keep-Alive Date: Wed, 22 Jun 2016 15:59:26 GMT {:swagger "2.0", :info {:title "Edge API", :version "1.0", :description "An example API"}, :produces ["application/json"], :consumes ["application/json"], :paths {"/hello" {:get {:produces ("text/plain"), :responses {:default {:description ""}}}}}, :basePath "/api", :definitions {}}
It’s these little details that yada takes care of for you. There is no trickery involved, it’s simply the result of an almost obsessive focus on the relevant web standards. There is nothing special about strings, yada applies the same logic for anything else you ask it to handle. We’ll see more in the next chapter.
By the way, if you want to see the Swagger UI, browse to http://localhost:3000/swagger/?url=http://localhost:3000/api/swagger.json
3.8. Summary
This has been a long chapter, but we have only covered a simple "Hello World!" example.
You should now realise that implementing even a basic example that properly complies with the HTTP standard is a surprisingly tough undertaking. But this simple example demonstrated how a rich and functional HTTP resource can be created with a tiny amount of code, and none of the behaviour we have seen is hardcoded or contrived. We have only demonstrated a simple Java string, and yada includes similar support for many other basic types (atoms, Clojure collections, files, directories, Java resources…).
But the best thing is you can programmatically create your own resources and types to fit your particular requirements.
Without a library like yada we would need to read and understand hundreds of pages of the HTTP RFCs and spend a great deal of extra effort coding up its various aspects. Of course, nobody would bother doing that, but the consequence is that we miss out on the many architectural benefits of HTTP.
Rarely can client code make any assumptions that the HTTP API it is accessing complies with the text in the HTTP RFCs, and must therefore rely on detailed knowledge of how the API is written, either through documentation, Swagger definitions, close collaboration between development teams or some other means (trial-and-error). This is a problem because it causes rigidity between our systems.
By using yada, we are pushing the responsibility of implementing HTTP correctly away from the programmer and into a library.
4. Installation
yada is a Clojure library and if you are using it directly from a
Clojure application with a Leiningen project.clj
file, include the
following in the file’s :dependencies section.
[yada "1.2.15"]
[aleph "0.4.1"]
[bidi "2.0.12"]
4.1. Setting up a development environment
The best way to learn is to experiment with yada, either on its own or by using it to build some service you have in mind. Either way you’ll want to set up a quick development environment.
4.2. The Easy Way: Clone edge
The quickest and perhaps easiest way to get started is to clone the yada branch of JUXT’s edge repository. This is a continuously improving development environment representing our firm’s best advice for building Clojure projects.
$ git clone git@github.com/juxt/edge
edge is opinionated and combines a number of practices that we’ve found useful on many projects at JUXT.
If you’re new to Clojure, we recommend learning yada with edge. Not only is it packed full of examples for you to explore, the project can provide a solid foundation for your own projects with an architecture that is proven to scale as your project grows.
4.3. The Simple Way: Construct your own
If you prefer to build your own development environment you’ll need a few pointers in order to integrate with yada.
4.3.1. Serving resources
To serve the resources you define with yada you’ll need to choose a port and start a web server. Currently, yada provides its own built-in web server (Aleph), but in the future other web servers will be supported. (For now, please be reassured that Aleph will support many thousands of concurrent connections. Aleph is built on Netty, a very capable platform used by Google, Apple, Facebook and many others to support extreme workloads.)
To start a web-server, just require listener
from yada.yada
and call it with a route structure and some configuration that includes the port number.
(require '[yada.yada :refer [listener resource as-resource]])
(def svr
(listener
["/"
[
["hello" (as-resource "Hello World!")]
["test" (resource {:produces "text/plain"
:response "This is a test!"})]
[true (as-resource nil)]]]
{:port 3000}))
The listener
function returns a map. To shutdown the listener, call the 0-arity function contained in the :close
entry.
((:close svr))
4.3.2. Creating resources
Resources can be created in the following ways:
-
From a resource model, calling
resource
with a map argument -
Using
as-resource
on an existing type known to yada -
A normal Ring handler
4.3.3. Routing to resources
A handler can be created from a resource by yada's handler
function, and this handler can be used anywhere a Ring-compatible function is expected. This way, you can use any routing library you wish, including Compojure.
However, yada and bidi are designed to work together, and there are a number of features that are enabled when they are used together. That’s why you can put yada resources in a bidi route structure without turning them into handlers, since bidi knows how to do that already.
5. Resources
In yada's terminology, a resource is the same as it is in HTTP:
The target of an HTTP request is called a "resource". HTTP does not limit the nature of a resource; it merely defines an interface that might be used to interact with resources. Each resource is identified by a Uniform Resource Identifier (URI),
5.1. Resource models
We can describe a resource directly using a plain old Clojure map called a resource model.
Here’s an example:
(require '[yada.yada :refer [resource]])
(def my-resource
(resource
{:id :example
:description "The description to this example resource"
:summary "An example resource"
:access-control …
:properties …
:parameters {:query … :path … :header …}
:produces …
:consumes …
:methods {:get … :put … :post … :delete … :patch …}
:responses {…}
:path-info? false
:sub-resource …
:logger …
:interceptor-chain …
:error-interceptor-chain …
:custom/other …}))
Resource models are constrained by a schema, ensuring they are valid. The purpose of the yada.yada/resource
function is to check the resource model is valid. The schema will attempt to coerce invalid resource models into their valid equivalents wherever possible. An error will be thrown only after these coercions have been attempted.
The result of a call to yada.yada/resource is actually an instance of the Clojure record yada.resource/Resource but you can treat it just like a map.
|
The following sections describe the anatomy of a resource model in more depth.
You can augment a resource model with your own data if you like, but the keys you use must be namespaced keywords. Don’t use the yada namespace as that’s reserved for future use (in a future version of yada, all its keywords will be namespaced with yada ).
|
5.1.1. Resource identity
The optional :id
entry of the resource model gives the resource a unique identity. You can use whatever you like for the value, but it should be unique. A namespaced keyword is typical:
{:id :resources/user-profile}
The main reason for giving your resources an identity is for creating hyperlinks targeting your resource. For example, this is how you would create a URL to the resource.
(yada.yada/path-for ctx :resources/user-profile)
An example with a parameter:
(yada.yada/path-for ctx :resources/user-profile
{:route-params {:user-id "42"}})
This feature is only available if your resources are declared in a bidi hierarchical route structure. Otherwise, the URL cannot be determined.
5.1.2. Resource description and summary
Optionally, a resource can contain a textual description. This should be used for any descriptive text that applies to the resource as a whole (rather than individual methods, which can contain their own descriptions).
{:description "<descriptive text here>"
:summary "<summary here>"}
The description and summary values are used in generated Swagger descriptions and can be used for any other purpose you like.
5.1.3. Security & Access Control
The :access-control
entry can be used to restrict access to a resource and provide security. It encompasses authentication and authorization, if necessary across multiple realms. It also determines the circumstances that the resource can be accessed from different origins for browser interaction (CORS) as well as defining other security protections.
Multiple authentication schemes are supported, such as Basic, JWT and OAuth2.
Access control and security are fully described in the security chapter.
5.1.4. Properties
You can define various properties on a resource. These can be thought of as a resource’s metadata, information about a resource (rather than the resource’s state).
It is possible to specify a complete map of constant properties, if they are all known prior to a request. This is rare, and usually it’s necessary to provide a function that will be called during the processing of a request.
{:properties (fn [ctx]
{:exists? true
:last-modified #inst "2016-07-25 16:00:00 Z"})}
Certain properties, such as :exists?
and :last-modified
are special and used by yada to determine responses.
For example, if you know how to determine the date that your resource was last modified, you should return this date in the :last-modified
entry of a map containing your resources’s properties. Doing so will enable yada's logic for conditional requests, for instance, allowing it to return 304 Not Modified
responses when appropriate.
More information can be found in Properties.
5.1.5. Parameters
Web requests can contain parameters that can influence the response and yada can capture these. This is especially useful when you are writing APIs.
There are different types of parameters, which you can mix-and-match:
-
Query parameters (part of the request URI’s query-string)
-
Path parameters (embedded in the request URI’s path)
-
Request headers
-
Form data
-
Request bodies
-
Cookies
There are benefits to declaring these parameters explicitly:
-
yada will check they exist, and return 400 (Malformed Request) errors on requests that don’t provide the ones you need for your logic
-
yada will coerce them to the types you want, so you can avoid writing loads of type-conversion logic in your code
-
yada and other tools can process your declarations independently of your request-processing code, e.g. to generate API documentation
Parameter declaration, validation and coercion is a big topic and fully covered in the parameters chapter.
5.1.6. Representations
Resources have physical forms called representations. A resource can declare all the representations it supports.
Typically, a representation will have a designated content type, such as text/html
or application/json
, which tells the receiver how to process it.
Example: The string "Hello World!" might have the type text/plain
. But the string "<h1>Hello World!</h1>" might be given the type text/html
to indicate that it should be rendered as HTML.
If the content type of a representation begins with text/
, it might also have a given charset, indicating how the bytes transferred should be turned into text.
Some representations will also indicate whether the content is compressed (called the encoding) and maybe the language used.
It is often useful to distinguish between outward representations that can be produced and the inward representations that can be consumed.
The :produces
entry in the resource model declares the representations of the resource that can be produced.
Where there is more than one representation that can be produced, yada negotiates which type, if any, is actually produced taking into account the declared preferences of the user agent. This process is known as content negotiation.
The :consumes
entry declares the incoming representations that the resource is able to accept. Some HTTP methods allow requests to contain bodies. Here there is no content negotiation, since the user agent will tell the server the content type of the body it is sending.
More details can be found in the representations chapter.
5.1.7. Methods
The :methods
entry is a map, where each key is a keyword that corresponds to an HTTP method.
{:methods
{:get {…}
:post {…}
:brew {…}}}
There is no restriction on the methods you can declare.
The value of each method entry is also a map, which has the following structure:
{:response (fn [ctx] …)
:parameters {…}
:produces {…}
:consumes {…}
:authorization {…}
:description ""
:summary ""
:responses {404 {:description "Not found"}}
:custom/other …}
Each method has a specific prescribed behaviour, called the method’s semantics, which usually described in a particular RFC document (but it’s also fine to define your own).
Method semantics for core HTTP methods are built-in to yada but it’s possible to add your own via a Clojure protocol.
Many method semantics will involve a call to the function you declare in the :response
entry, which is responsible for constructing the response, but if you’re not sure you should check the description for the actual method you’re using in the methods chapter.
5.1.8. Responses
By default, yada will produce error messages and stack traces for various status codes. If you wish to override this behaviour, you must provide alternatives via the :responses
entry of the resource map.
For example, perhaps you want to provide a particular response that is generated whenever there is a 404 Not Found error. Many websites like to do this, perhaps as a hint to the user to check the URL.
In the response map we would add something like this:
{:responses
{404 {:produces #{"text/html"}}
:response (fn [ctx] …)}}
5.1.9. Path info
The path-info?
entry is a boolean flag which indicates whether the resource expects a path-info.
Imagine your URI path is /dir/abc/foo.txt
. You may want to partially match this path such that the resource is called for all URIs that begin with /dir/
. In this case, abc/foo.txt
would be set as the path-info in the request map.
The reason we might want to indicate this on the resource is to tell our router that a partial match is required, and to give us access to the remaining path.
5.1.10. Sub-resources
Sometimes we cannot know the properties of a given resource up-front. For example, imagine you are serving files from a file-system. It is impossible to determine which resources will be present when the request arrives, and therefore which properties and content attributes should be declared.
To support such dynamic resources, yada allows the declaration of a function, as the value of the :sub-resource
key, that will be called when the request arrives. The return value of the sub-resource function must return the actual resource.
This feature is commonly used together with path-info to provide dynamic groups of related resources.
5.1.11. Logging
The :logger
entry can declare a function which is called whenever a request is processed and the response is about to be returned to the web-server. This allows you to log all requests to a file, for instance.
5.1.12. Interceptor chains
yada is built on a chain of interceptors that are processed asynchronously. For most cases, the default interceptor chain will suffice, but sometimes it is necessary to add to this chain, or modify it in some way, on a resource-by-resource basis. This is achieved by providing an alternative interceptor chain via the :interceptor-chain
and :error-interceptor-chain
entries.
5.2. Resources as Ring handlers
Now we have introduced all the entries that a resource model can contain, let’s use our knowledge to re-create a basic "Hello World!" resource:
(require '[yada.yada :as yada])
(def my-resource
(yada/resource
{:produces {:media-type "text/plain"}
:methods {:get
{:response (fn [ctx] "Hello World!")}}}))
Now we have a valid resource, we can now use it for a range of purposes — one obvious one is to handle HTTP requests. We can create a Ring request handler from a resource with the yada.yada/handler
function:
(def my-ring-handler
(yada/handler my-resource))
We can now use this handler in a route.
For example, with Compojure:
(GET "/my-resource" [] my-ring-handler)
Or with bidi:
["/my-resource" my-ring-handler]
Note, since yada is aware of bidi’s
|
5.2.1. Responding to requests
The handler created by yada works by constructing a series of internal functions called interceptors.
When a request is received, the handler creates a new instance of an object known as the request context, and its idiomatic symbol is ctx
.
Each interceptor is a single-arity function that takes this request context as an argument, returning the same request context or a modified copy.
Here’s an interceptor which adds some information into the request context:
(fn my-interceptor [ctx]
(assoc ctx :film "Life Of Brian"))
On each request, the request context is threaded through a chain of interceptors, the result of each interceptor being used as the argument to the next.
One of the entries in the request context is :response
, which contains the Ring response that will be returned to the web server. Any interceptor can modify this (or any other value) in the request context.
Here’s an example of a request context during the handling of a request:
{:request {:method :get :headers {…}}
:request-id #uuid "bf2c06e1-b4bd-49fb-aa74-05a17f4e9e9c"
:method :get
:response {:status 200 :headers {} :body "Hello!"}}
The request context is not just passed to interceptors, but to functions you can declare in your resource.
5.3. Resource types
A resource type is a Clojure type or record that can be automatically coerced into a resource model. These types must satisfy the yada.protocols.ResourceCoercion
protocol, and any existing type or record may be extended to do so, using Clojure’s extend-protocol
macro.
(extend-type datomic.api.Database
yada.protocols/ResourceCoercion
(as-resource [_]
(resource
{:properties
{:last-modified …}
:methods
{:get …}}})))
The as-resource
function must return a resource (by calling yada.resource/resource
, not just a map).
6. Parameters
As we learned in Parameters, parameters declarations are useful to validate and coerce request parameters, produce 400 status responses on bad requests thereby providing some protection against bad input data.
Many requests embed parameters in their URIs. For example, let’s imagine a URI to access the transactions of a fictitious bank account.
https://bigbank.com/accounts/1234/transactions?since=tuesday
There could be 2 parameters here. The first, 1234
, is contained in the
path /accounts/1234/transactions
. We call this a path parameter.
The second, tuesday
, is embedded in the URI’s query-string (after the
?
symbol). We call this a query parameter.
You can declare these parameters in the resource model.
{:parameters {:path {:entry Long}}
:methods {:get {:parameters {:query {:since String}}}
:post {:parameters {:body …}}}
Parameters can be specified at resource-level or at method-level. Path parameters are usually declared at the resource-level because they form part of the URI that is independent of the request’s method. In contrast, query parameters usually apply to GET requests, so it’s common to define this parameter at the method-level, and it’s only visible if the method we declare it with matches the request method.
We declare parameter values using the syntax of Plumatic's schema library. This allows us to get quite sophisticated in how we define parameters.
(require [schema.core :refer (defschema)]
(defschema Transaction
{:payee String
:description String
:amount Double}
{:parameters {:path {:entry Long}}
:methods {:get {:parameters {:query {:since String}}}
:post {:parameters {:body Transaction}}}
6.1. Capturing multi-value parameters
Occasionally, you may have multiple values associated with a given parameter. Query strings and HTML form data both allow for the same parameter to be specified multiple times.
/search?accno=1234&accno=1235
To capture all values in a vector, declare your parameter type as a vector type:
{:parameters {:query {:accno [Long]}}}}
6.2. Capturing large request bodies
Sometimes, request bodies are very large or even unlimited. To ensure you don’t run out of memory receiving this request data, you can specify more suitable containers, such as files, database blobs, Amazon S3 buckets or your own extensions.
All data produced and received from yada is handled efficiently and asynchronously, ensuring that even with very large data streams your service continues to work.
{:parameters {:form {:video java.io.File}}}
7. Properties
Properties tell us about the current state of a resource, such as whether the resource exists, or when the resource was last modified. Properties allow us to determine whether the user agent’s cache of a resource’s state is up-to-date.
Sometimes all a resource’s properties are constant and can be known when the resource is defined. More likely the resource’s properties have to be determined by some logic, and often this logic involves I/O.
Also, if the resource has declared parameters, it can be that the resource’s properties depend in some way on these parameters. For example, the properties of account A may well be different from the properties of account B.
A resource’s properties may also depend on who is making the request. Your bank account details should only exist if you’re the one accessing them. If I tried to access your bank account details, you’d want the service to behave differently.
For this reason, a resource’s properties declaration in the resource-model points to a single-arity function that is called by yada after the request’s parameters have been parsed and the credentials of a caller have been established.
In many cases, it will be necessary to query a database, internal web-service or equivalent operation involving I/O.
If you use a properties function, anything you return will be placed in the :properties entry of the request-context. Since the request-context is available when the full response body is created, you may choose to return the entire state of the resource, in addition to its properties. This may be sensible if it helps avoid a second trip to the database.
8. Methods
Methods are defined in the methods entry of a resource’s resource-model.
Only methods that are known to yada can appear in a resource’s
definition. Each method corresponds to a type that extends the
yada.methods.Method
protocol. This design also makes it possible to
add new methods to yada, as required.
The responsibility of each method type is to encode the semantics of the corresponding method as it is defined in HTTP standards, such as responding with the correct HTTP status codes (most other web frameworks delegate this responsibility to developers).
Some methods can be implemented entirely by yada itself (HEAD, OPTIONS, TRACE etc.). Most methods, however, delegate to some function or functions declared in the method’s declaration in the resource-model.
8.1. Method semantics, by method
Each HTTP method has defined semantics. Often these semantics are defined in the HTTP standards, other RFCs or, in the case of custom methods, by you.
These semantics are important because they allow other web agents, such as browsers and proxies, to inter-operate with your site.
Below is an explanation of the semantics for every method yada currently supports and your responsibilities should you choose to provide the method for a resource.
8.2. GET
Specify a function in :response that will be called during the GET method processing.
If the resource exists, the single-arity function will be called with the request context as its only argument.
It should return the response’s body, which should satisfy
yada.body.MessageBody
determining how exactly the response’s body will
be returned.
8.10. Handling all methods
If you want to handle all request methods, or a complex set of them, you
can specify the special keyword :*
in the methods section of your
resource model.
{:methods
{:*
{:response (fn [ctx] …)}}}
8.11. Custom methods
Custom methods can be added by defining new types that extend the
yada.methods.Method
protocol.
8.12. BREW
BREW is an example of a custom method you might want to create, especially if you are building a networked coffee maker compliant with RFC.
(require '[yada.methods Method])
(deftype BrewMethod [])
(extend-protocol Method
BrewMethod
(keyword-binding [_] :brew)
(safe? [_] false)
(idempotent? [_] false)
(request [this ctx] …))
9. Representations
Resources have state, but when this state needs to be transferred from one host to another, we use one of a number of formats to represent it. We call these formats representations.
A given resource may have a large number of actual or possible representations.
Representation may differ in a number of respects, including:
-
the media-type (file format)
-
if textual, the character set used to encode it into octets
-
the (human) language used (if textual)
-
whether and how the content is compressed
Whenever a user-agent requests the state from a resource, a particular representation is chosen, either by the server (proactive) or client (reactive). The process of choosing which representation is the most suitable is known as content negotiation.
9.1. Producing content
Content negotiation is an important feature of HTTP, allowing clients and servers to agree on how a resource can be represented to best meet the availability, compatibility and preferences of both parties. It is a key factor in the survival of services over time, since both new and legacy media-types can be supported concurrently. (It is also the mechanism by which new versions of media-types can be introduced, even media-types that define hypermedia interactions, more on this later.)
9.2. Proactive negotiation
There are 2 types of content negotiation. The first is termed proactive
negotiation where the server determines the type of representation from
requirements sent in the request headers. These are the headers
beginning with Accept
.
For any resource, the available representations that can be produced by a resource, and those that it consumers, are declared in the resource model. Every resource that allows a GET method should declare at least one representation that it is able to produce.
Let’s start with a simple web-page example.
{:produces "text/html"}
This is a short-hand for writing […]
(missing text here)
clojure {:produces [{:media-type "text/plain" :language #{"en" "zh-ch"} :charset "UTF-8"} {:media-type "text/plain" :language "zh-ch" :charset "Shift_JIS;q=0.9"}]
( todo - languages )
$ curl -i http://localhost:8090/hello-languages -H "Accept-Charset: Shift_JIS" -H "Accept: text/plain" -H "Accept-Language: zh-CH"
HTTP/1.1 200 OK Content-Type: text/plain;charset=shift_jis Vary: accept-charset, accept-language, accept Server: Aleph/0.4.0 Connection: Keep-Alive Date: Mon, 27 Jul 2015 18:38:01 GMT Content-Length: 9 ?�D���E!
9.3. Reactive negotiation
The second type of negotiation is termed reactive negotiation where the agent chooses from a list of representations provided by the server.
(Currently, yada does not yet support reactive negotiation but it is definitely on the road-map.)
9.5. Body coercion
Where necessary, and according to the semantics of the HTTP method, yada will coerce the result of a method into an entity body of the negotiated content type.
For example, consider the following resource-map.
clojure {:produces "application/json" :methods {:get {:response (fn [_] {:greeting "Hello"})}}}
On receiving a GET request, yada will automatically convert the map
{:greeting "Hello"}
into the JSON body {"greeting":"Hello"}
.
However, the key phrase here is where necessary. If a string is
returned from a method, and the content type is application/json
,
there is some ambiguity between whether the resource developer intends
for yada to encode the string into JSON, or whether the string is
already JSON encoded. In these ambiguous cases, yada assumes the
string is JSON encoded already.
Therefore this code would produce an error when the user agent attempts to decode the JSON string.
{:produces "application/json"
:methods
{:get
{:response (fn [_] "This is not JSON")}}}
If you are faced with this situation, you should choose one of the following options:
-
Return a map with the JSON string embedded. For example:
{:message "Plain old string"}
. -
Use a JSON encoder such as
org.clojure/data.json
or Cheshire and encode the string yourself.
10. Responses
In resource interactions, a request is processed by a method, usually resulting in a response. The response contains a status code, headers and a body.
yada attempts to set the correct status code and headers according to the semantics of both the method and mime-type. It also coerces the result of a method into the response body if necessary.
Sometimes, however, the response returned by yada is not what you want. There are times that you want more fine-grained control, want to provide custom bodies for certain status codes (such as 404 errors), or want deviate from the HTTP standards entirely.
10.1. Explicit responses
When you need complete control over the response you should return the request-context's response, modified if necessary. In which case yada will see that you want to be explicit and get out of your way.
(fn [ctx]
(let [response (:response ctx)]
;; return a response, explicitly associating
;; (or updating) the status, headers or body.
(update-in response […] …)
))
10.2. Declared responses
yada declares the responses that may normally be produced by a method. It adds these declarations to the resource-model when creating a resource prior to processing.
However, with explicit responses you may generate response codes that are unexpected by yada and which could be declared through the resource-model.
In these cases, you should declare the response codes in the :responses entry of the resource’s resource-model.
{:responses
{418 {:description "I'm a teapot"}}}
The keys in the :responses map can be integers, sets of integers, or
the wildcard: *
. Only sets are supported, so if you need to produce a
range of status codes, create a set programmatically:
{(set (concat (range 400 404) (range 405 500))}
{:description "All errors besides 404"
…
}}
Individual status codes take precedence over sets of status codes, which take precedence over wildcards. After that, the order in which keys are checked is undefined. If you are using sets, avoid declaring the same status code in multiple keys.
10.3. Status responses
Usually yada will return the responses produced by methods, and create ones for errors that occur along the way.
Often it is useful to be able to control the response body, or even the complete response, for responses with certain status codes.
We can specify these controlled responses by specifying a response entry in the value corresponding to the status code.
A common example is providing a custom 404 page when a resource cannot be found, which may provide the user with details of why the resource couldn’t be found and perhaps what to do next.
Let’s see how this is done:
(require '[yada.yada :as yada])
{:properties {:exists? false}
:responses
{404 {:description "Not found"
:produces #{"text/html" "text/plain;q=0.9"}
:response (let [msg "Oh dear I couldn't find that"]
(fn [ctx]
(case (yada/content-type ctx)
"text/html" (html [:h2 msg])
(str msg \newline))))}}}
Note that the response definition can include a declaration of the representations that the response can produce.
If the response is caused by an error, the actual error is available in the context under :error.
11. Security
Built-in to the library, yada offers a complete standards-based set of security features for today’s secure applications and content delivery.
11.1. Security is part of the resource, not the route
In yada, resources are self-contained and are individually protected from unauthorized access. We agree with the HTTP standards authors when we consider security to be integral to the definition of the resource itself, and not an extra to be bolted on afterwards. Nor should it be complected with routing. The process of identifying up a resource from its URI is independent of how that resource should behave, and shouldn’t be coupled to it.
Building security into each resource yields other benefits, such as making it easier to test the resource as a unit in isolation of other resources and the router.
As in all other areas, yada aims for 100% compliance with core HTTP standards when it comes to security, notably RFC 7235. Also, since HTTP APIs are nowadays used to facilitate transactional integration between systems via the user’s browser, it is critically important that yada fully supports systems that offer APIs to other applications, across origins, as standardised by CORS.
11.2. The :access-control
entry
All security aspects for a resource are specified in its model’s
:access-control
entry.
11.3. Authentication
Let’s look at authentication first. Authentication is the process of establishing and verifying the identity and credentials of the user, with reasonable confidence that the user is not an impostor.
In yada, authentication happens after any request parameters have been processed, so if necessary they can be used to establish the identity of the user. However, it is important to remember that authentication happens before the resource’s properties have been loaded, since credentials do not have to do with the actual resource. Thus, if the user is not genuine, we might well save a wasted trip to the resource’s data-store. |
In HTTP, resources can exist inside a protection space determined by one or more realms. Each resource declares the realm (or realms) it is protected by, as part of the :access-control entry of its resource-model.
11.3.1. Authentication schemes
Each realm declares one or more authentication schemes governing how requests are authenticated.
yada supports numerous authentication schemes, including custom ones you can provide yourself.
Each scheme has a verifier. Depending on the scheme this is usually a function. The verifier is used to extract or otherwise establish the credentials in the request, ensuring they are authentic and true, in which case it returns these credentials as a value to be stored in the request context. These credentials may contain information such as the user’s identity, roles and privileges. If no credentials are found, the verifier should return nil.
If no credentials are found by any of the schemes, a 401 response is returned containing a WWW-Authenticate header.
11.3.2. Basic authentication
Here is an example of a resource which uses Basic authentication described in RFC 2617
{:access-control
{:realm "accounts"
:scheme "Basic"
:verify (fn [[user password]] …)}}
There are 3 entries here. The first specifies the realm, which is
defaults to default
in Basic Authentication, but if specified is
contained in the dialog the browser presents to the user.
The second declares we are using Basic authentication.
The last entry is the verify function. In Basic Authentication, the verify function takes a single argument which is a vector of two entries: the username and password.
If the user/password pair correctly identifies an authentic user, your function should return credentials.
(fn [[user password]]
…
{:email "bob@acme.com"
:roles #{:admin}})
If the password is wrong, you may choose to return either an empty map or nil. If you return an empty map (a truthy value) and the resource requires credentials that aren’t in the map, a 403 Forbidden response will be returned. However, if you return nil, this will be treated as no credentials being sent and a 401 Unauthorized response will be returned.
From a UX perspective there is a difference. If the user-agent is a browser, returning nil will mean that the password dialog will reappear for every failed login attempt. If you return truthy, it will show the 403 Forbidden response.
You may choose to limit the number of times a failed login attempt is tolerated by setting a cookie or other means.
11.3.4. Cookie authentication
We can also use cookies to present authentication credentials. The advantage of cookies is that they can be set by the server based on custom authentication interaction with the user, such as the submission of a login-form.
To protect a site with cookies:
{:access-control
{:scheme :cookie
:cookie "session"
:verify (fn [cookie] …}}
11.3.6. Form-based logins
Basic Authentication has a number of weaknesses, such as the difficulty of logging out and the lack of control that a website has over the fields presented to a human. Therefore, the vast majority of websites prefer to use a custom login form generated in HTML.
You can think of a login form as a resource that lets the user present one set of credentials in order to acquire additional ones. The credentials the user presents, via a form, are verified and if they are true, a cookie is generated that certifies this. This cookie provides the certification to subsequent requests in which it is sent.
Let’s start by building this login resource that will provide a login form page to browsers and verify the form data when that form is submitted.
Here’s a simplistic but viable resource model for the two methods involved:
(require
'[buddy.sign.jwt :as jwt]
'[schema.core :as s]
'[hiccup.core :refer [html])
{:methods
{:post
{:consumes "application/x-www-form-urlencoded"
:parameters {:form
{:user s/Str :password s/Str}}
:response
(fn [ctx]
(let [{:keys [user password]} (get-in ctx [:parameters :form])]
(if (valid-user user password)
(assoc (:response ctx)
:cookies {"session"
{:value
(jwt/sign {:user user} "lp0fTc2JMtx8")}})
"Try again!")))}
:get
{:produces "text/html"
:response (html
[:form {:method :post}
[:input {:name "user" :type :text}]
[:input {:name "password" :type :password}]
[:input {:type :submit}]])}}}
The POST method method consumes incoming URL-encoded data (the classic way a browser sends form data). It de-structures the two parameters (user and password) from the form parameters.
We then determine if the user and password are valid (we don’t explain
here how this is done, but assume a valid-user
function exists that
can tell us). If the user is valid we associate a new cookie called
"session" with the response. By starting with the :response
value of
the request context, we ensure yada interprets our return value as a
Ring response rather than some other value.
We use Buddy’s sign
function to sign and encoded the cookie’s value as
a JSON string. We only specify the credentials as {:user user}
in this
case, but we could put much more into that map. The sign
function
requires us to provide a secret symmetric key that we can use for both
signing and verification, but the library does allow us asymmetric key
options too.
The other method, GET, simply produces a form for user-agents that can render HTML (browsers, typically) to post back. For reasons of cohesion, it’s a good idea to provide these two methods in the same resource to encapsulate and dedupe the fields which are relevant to both the GET and the POST.
11.4. Authorization
Authorization is the process of allowing a user access to a resource. This may require knowledge about the user only (for example, in Role-based access control). Authorization may also depend on properties of the resource identified by the HTTP request’s URI (as part of an Attribute-based access control authorization scheme).
In either case, we assume that the user has already been authenticated, and we are confident that their credentials are genuine.
In yada, authorization happens after the resource’s properties have been loaded, because it may be necessary to check some aspect of the resource itself as part of the authorization process.
By default, yada will use a declarative role-based authorization scheme.
11.4.1. Default authorization scheme
Any method can be protected by declaring a role or set of roles in its model.
{:access-control
{:authorization
{:methods
{:post :accounts/create-transaction}}}}
If multiple roles are involved, they can be composed inside vectors using simple predicate logic.
{:access-control
{:authorization
{:methods
{:post [:or [:and :accounts/user
:accounts/create-transaction]
:superuser}}}}
Only the simple boolean operators of :and
, :or
and :not
are
allowed in this authorization scheme. This keeps the role definitions
declarative and easy to extract and process by other tooling.
Of course, authentication information is available in the request context when a method is invoked, so any method may apply its own custom authorization logic as necessary. However, yada encourages developers to adopt a declarative approach to resources wherever possible, to maximise the integration opportunities with other libraries and tools.
11.4.2. Custom authorization scheme
A custom authorization scheme can be declared that will completely replace the default authorization scheme already discussed.
First, decide on a keyword that will be used to dispatch your
authorization function. In this example, we’ve chosen
:my/custom-authorization
.
Now declare the authorization function that will be called by yada
during request processing. This is a defmethod
, as follows:
(defmethod yada.authorization/validate
:my/custom-authorization
[ctx credentials authorization]
…
)
The credentials argument contains all the verified credentials sent in the request.
Now add an :authorization
map to the :access-control
part of your
resource model. The map must contain a :scheme
value specific to your
resource model, along with any extra parameters you want to be passed as
the authorization
argument to your authorization function. In this
example, we want to pass the :my/ensure
parameter set to
[:same-account]
. You can specify anything you like to be passed as
parameters (there are no schema restrictions here).
{:access-control
{:authorization
{:scheme :my/custom-authorization
:my/ensure [:same-account]}}}
11.5. Realms
yada supports multiple realms. By default, there is a single realm in operation called "default". However, you can group authentication schemes and authorization models in separate realms. Each realm can contain multiple authentication schemes (it might be that a realm offers a choice of how to authenticate).
{:access-control
{:realms {"Gondor" {:authentication-schemes […]
:authorization {…}}
"Mordor" {:authentication-schemes {…}
:authorization {…}}}}}
11.6. Cross-Origin Resource Sharing (CORS)
yada fully supports Cross-Origin Resource Sharing (CORS) allowing you to provide APIs that are accessible from other origins.
For example, you may be creating an API that you wish other websites to make use of, by allowing browsers visiting those websites access to your API.
CORS is specified in the :access-control
section of the
resource-model.
{:access-control
{:allow-origin "*"
:allow-credentials false
:expose-headers #{"X-Custom"}
:allow-methods #{:get :post}
:allow-headers ["Api-Key"]
}}
With the exception of :allow-credentials
(which must be a boolean),
any of the values can be declared as single-arity functions, which are
called with the request-context as an argument to determine the value
for the corresponding response header.
11.7. HTTP Strict Transport Security (HSTS)
clojure {:strict-transport-security {:max-age 12000}}
Defaults to a maximum age of 31536000.
The HSTS header is only set if the scheme is HTTPS or the service is
behind a proxy (determined by the presence of the X-Forwarded-For
request header).
11.8. Content Security Policy
{:content-security-policy "url-src"}
Defaults to default-src https: data: 'unsafe-inline' 'unsafe-eval'
.
11.9. Clickjacking prevention
A browser’s iframe can be used for click-jacking. By default yada
tells browsers not to allow this. The default value is SAMEORIGIN
,
unless you override it in the resource-model.
{:x-frame-options "NONE"}
11.10. Cross-site Scripting (XSS) protection
yada also sets the X-Xss-Protection
response header to
1; mode=block
. This can be overridden in the resource model.
{:x-content-type-options "0"}
11.11. Media-type sniffing protection
By default, yada sets the X-Content-Type-Options
response header to
nosniff
. This tells browsers not to try to attempt to determine the
content-type of the response body.
Since yada sets the Content-Type
header according to HTTP standards,
there should never be a need for a browser to sniff the response body
for this information, preventing an attack that might exploit some
vulnerability in this process.
12. Routing
Since the yada
function returns a Ring-compatible handler, it is
compatible with any Clojure URI router.
However, yada is designed to work especially well with its sister library bidi, and unless you have a strong reason to use an alternative routing library, you should stay with the default.
While yada is concerned with semantics of how a resource responds to requests, bidi is concerned with the identification of these resources. In the web, identification of resources is a first-class concept. Each resource on the web is uniquely identified with a Uniform Resource Identifier (URI).
No resource is an island, and it is common that resource representations need to embed references to other resources. This is true both for ad-hoc web applications and hypermedia APIs, where the client traverses the application via a series of hyperlinks.
These days, hyperlinks are so critical to the reliable operation of systems that it is no longer satisfactory to rely on ad-hoc means of constructing these URIs, they must be generated from the same tree of data that defines the API route structure.
12.1. Declaring your website or API as a bidi/yada tree
A bidi routing model is a hierarchical pattern-matching tree. The tree is made up of pairs, which tie a pattern to a (resource) handler (or a further group of pairs, recursively).
Both bidi’s route models and and yada's resource models are recursive data structures and can be composed together.
The end result might be a large and deeply nested tree, but one that can be manipulated, stored, serialized, distributed and otherwise processed as a single dataset.
(require
[yada.yada :refer [resource]]
[hiccup.core :refer [html]]
[clojure.java.io :refer [file])
;; Our store's API
["/store/"
[ ; Vector containing our store's routes (bidi)
["index.html"
{:summary "A list of the products we sell"
:methods
{:get
{:response (file "index.html")
:produces "text/html"}}}]
["cart"
{:summary "Our visitor's shopping cart"
:methods
{:get
{:response (fn [ctx] …)
:produces #{"text/html" "application/json"}}
:post
{:response (fn [ctx] …)
:produces #{"text/html" "application/json"}}}}]
…
]]
A yada handler (created by yada’s yada
function) and a
yada resource (created by yada’s resource
constructor
function) that extends bidi’s Matched
protocol are both able to
participate in the pattern matching of an incoming request’s URI.
For a more thorough introduction to bidi, see https://github.com/juxt/bidi/blob/master/README.md.
12.2. Declaring policies across multiple resources
Many web frameworks allow you to set a particular behavioral policy (such as security) across a set of resources by specifying it within the routing mechanism.
In our view, this is wrong, for many reasons. A URI is purely a identifier for a resource. A resource’s identifier might change, but such a change should not cause the resource to bahave differently.
In the phraseology offered by Rich Hickey in his famous Simple Made Easy talk, we should not complect a resource’s identification with its operation. Neither should we complect a protection space with a URI space. Doing so adds an unnecessary constraint to the already difficult problem of naming things (URIs) while adding an unnecessary constraint to the ring-facing of resources into protection spaces. For this reason, yada and bidi are kept apart as separate libraries.
Some web frameworks can be excused for offering a pragmatic way of reducing duplication in specification, but this really ought not to be necessary for Clojure programmers who have powerful alternatives.
What are these alternatives? How can we avoid typing the same declarations over and over in every resource?
One option is to create a function that can augment a set of base resource models with policies. That function can then be mapped over a number of resources.
A variation of this option is to use Clojure’s built-in tree-walking
functions such as clojure.walk/postwalk
. If you specify your entire
API as a single bidi/yada tree, it is easy to specify each policy as a
transformation from one version of the tree to another. What’s more, you
will be able to check, debug and automate testing on the end result
prior to handling actual requests.
13. Example 2: Phonebook
We have covered a lot of ground so far. Let’s consolidate our knowledge by building a simple application, using all the concepts we’ve learned so far.
We’ll build a simple phonebook application. Here is a the brief:
13.1. Phonebook requirements
Create a simple HTTP service to represent a phone book.
Acceptance criteria. - List all entries in the phone book. - Create a new entry to the phone book. - Remove an existing entry in the phone book. - Update an existing entry in the phone book. - Search for entries in the phone book by surname.
A phone book entry must contain the following details: - Surname - Firstname - Phone number - Address (optional)
13.2. The database
Create a new namespace called phonebook.db
.
(ns phonebook.db)
We’ll create a database constructor, and some functions to access its contents.
This constructor creates a map with two entries, both refs. We could use an atom, but refs offer more flexibility.
(defn create-db [entries]
{:phonebook (ref entries)
:next-entry (ref (inc (apply max (keys entries))))})
Now let’s add some code to add an entry.
(defn add-entry [db entry]
(dosync
(let [nextval @(:next-entry db)]
(alter (:phonebook db) conj [nextval entry])
(alter (:next-entry db) inc)
nextval)))
13.3. Creating new phonebook entries
For this requirement, we are going to support the POST method.
Let’s add the following entry to the static properties of the
IndexResource
.
{:parameters {:post {:form {:surname String
:firstname String
:phone [String]}}}
…
}
This declaration tells yada what parameters we are expecting in the POST method. In return, yada will do the following:
-
Validate the request, ensuring that all the mandatory parameters have been provided
-
Coerce the parameters to the desired types (if possible).
-
Return a 400 response (if not).
-
Parse the request body.
-
Help prevent XSS scripting attacks, by ensuring that no unexpected parameters are allowed to pass through. We should still be careful of String parameters though.
14. Swagger
All yada resources are built on data which can be published in a variety of formats. A popular format is Swagger, which allows APIs to be quickly documented. This is particularly useful when multiple teams of developers need to share their service documentation with others during development.
Swagger involves the creation of JSON-formatted specifications which, in the absence of libraries like yada, are hand-authored. There exist a variety of code generation libraries that can take hand-authored specifications and generate code in various programming languages. However, a key disadvantage with code-generation approaches like this is they do not support round-trip engineering, that is, the steady iterative co-evolution of the specification with the code.
Since Swagger covers both URI routing as well as resources, we must involve routing information in the set of resources we wish to publish. Currently, bidi is the only supported router, but it should be possible to support other data-driven routers such as Silk in future.
The first task is to create and publish the swagger specification for a set of resources.
14.1. Creating the specification: the easy way
The easiest way of creating a Swagger spec is by following these steps:
14.1.1. Step 1:
Creating a bidi route structure containing your yada handlers (remember a yada handler is a Ring handler). Remember, bidi is infinitely recursive, so you can group your resources however you like. Just use the vector-of-vectors syntax in place of a usual handler.
["/greetings"
[
["/hello" (yada "Hello World!\n")]
["/goodbye" (yada "Goodbye!\n")]
]
]
14.1.2. Step 2: Wrap the route structure in swaggered
The swaggered
function can be used to further wrap the route
structure.
This function takes 2 arguments. The first argument is simply the bidi routing tree containing your yada resources. The second argument is a base template map which contains all the static data that should appear in the spec, such as the Swagger service meta-data. The data contained in both the routing tree and the resources themselves is used to construct the specification.
For example, let’s take the bidi routing tree below:
(require '[yada.swagger :refer [swaggered]])
["/api"
(swaggered
["/greetings"
[
["/hello" (yada "Hello World!\n")]
["/goodbye" (yada "Goodbye!\n")]
]
]
{:info {:title "Hello World!"
:version "1.0"
:description "A greetings service"}
:basePath "/api"}
)]
This declares a route that (partially) matches on the URI path /api
.
The second element of the route pair is a custom record created with
swaggered
which satisfies bidi’s Matched
protocol. This acts as a
sort of junction which matches on the routes given in the second
argument. Critically, however, it also adds an additional sub-route,
/swagger.json
, which exposes the Swagger specification of the given
base template, routes and resources.
Therefore, to access the JSON swagger specification in the example
above, you would navigate to /api/swagger.json
. Also, use
/api/greetings/hello
and /api/greetings/goodbye
to access the
services.
14.1.3. Creating the specification: the simple way
The Swagger spec is merely a map that can be created with
yada.swagger/swagger-spec-resource
. Once you have this map, publish it
with yada
(you should know how to do this already. Hint: (yada m)
).
There is a function yada.swagger/swagger-spec-resource
that creates a
yada resource for you, and can optionally take a content-type to publish
the spec in HTML and EDN too.
This approach gives you more flexibility, since you aren’t tied to publishing your swagger spec in the same route structure as your API (you might want to publish it on another server perhaps).
14.2. The Swagger UI
Once you have published the Swagger specification you should use the Swagger UI to access it.
If you use the swaggered
convenience function, a Swagger UI will
automatically be hosted under the route. Use a single /
to redirect to
the UI for the Swagger specification of your routing tree. In the
example above, navigating to /api/
will bring up the Swagger UI
allowing you to browse and play with the API.
It is also possible to host your own Swagger UI and link it to your
published Swagger specifications. Just pass the url
query parameter to
the Swagger UI to indiciate the location of the yada-produced Swagger
specification you want to browse.
Advanced users: For an example of custom Swagger UI configuration see
dev/resources/swagger/phonebook-swagger.html
for an example.
14.3. Resource options
Resources accept the following Swagger options. These options will also affect the Swagger UI.
-
:swagger/tags
for logical grouping of operations, e.g.["users"]
-
:swagger/summary
and:swagger/description
for documentation of operations
These options can be provided at the resource’s top level and can be overriden per method.
14.4. Swagger and REST
In some sense Swagger definitions compete with REST as an architecture. Where REST encourages self-describing APIs, Swagger tends towards well-documented APIs. REST APIs are particularly well suited to fluid public APIs which can support gradual evolution and a diverse and potentially unknown set of clients. In contrast, Swagger APIs and suited to fixed private APIs inside a single organisation or between multiple collaborating parties.
14.5. Data descriptions of APIs
Although the REST approach avoids publishing full API specifications up-front, preferring discovery over documentation, there are still many situations where it is useful to derive a data representation of an API.
One example is for API deployment to Amazon Web Services, where an API on the cloud can be created programmatically.
14.6. References
See IBM’s Watson Developer Cloud for a sophistated Swagger example.
Advanced topics
15. Async
Under normal circumstances, with Clojure running on a JVM, each request can be processed by a separate thread.
However, sometimes the production of the response body involves making requests to data-sources and other activities which may be I/O-bound. This means the thread handling the request has to block, waiting on the data to arrive from the I/O system.
For heavily loaded or high-throughput web APIs, this is an inefficient use of resources. Today, this problem is addressed by asynchronous I/O programming models. The request thread is able to make a request for data via I/O, and then is free to carry out further work (such as processing another web request). When the data requested arrives on the I/O channel, a potentially different thread carries on processing the original request.
As a developer, yada gives you fine-grained control over when to use a synchronous programming model and when to use an asynchronous one.
15.1. Deferred values
A deferred value is simply a value that may not yet be known. Examples include Clojure’s futures, delays and promises. Deferred values are built into yada — for further details, see Zach Tellman’s manifold library.
In almost all cases, it is possible to return a deferred value from any of the functions that make up our resource record or handler options.
For example, suppose our resource retrieves its state from another internal web API. This would be a common pattern with µ-services. Let’s assume you are using an HTTP client library that is asynchronous, and requires you provide a callback function that will be called when the HTTP response is ready.
On sending the request, we could create a promise which is given to the callback function and returned to yada as the return value of our function.
Some time later the callback function will be called, and its implementation will deliver the promise with the response value. That will cause the continuation of processing of the original request. Note that at no time is any thread blocked by I/O.
Actually, if we use Aleph or http-kit as our HTTP client the code is even simpler, since both libraries return promises from their request functions.
(require '[yada.yada :refer [resource]]
'aleph.http :refer [get])
(resource
{:methods
{:get (fn [ctx] (get "http://users.example.org"))}})
In a real-world application, the ability to use an asynchronous model is very useful for techniques to improve scalability. For example, in a heavily loaded server, I/O operations can be queued and batched together. Performance may be slightly worse for each individual request, but the overall throughput of the web server can be significantly improved.
Normally, exploiting asynchronous operations in handling web requests is difficult and requires advanced knowledge of asynchronous programming techniques. In yada, however, it’s easy, thanks to manifold.
For a fuller explanation as to why asynchronous programming models are beneficial, see the Ratpack documentation. (Note that yada provides all the features of Ratpack and more).
17. Server Sent Events
17.1. Introduction
Server Sent Events (SSE) is a part of the HTML5 generation of specifications that describes a capability for delivering events, asynchronously, from a server to a browser or other user-agent, over a long-lived connection.
SSE conceptually similar to web sockets. However, a key difference is that SSE is layered upon HTTP and thus inherits the protocol’s support for proxying, authorization, cookies and is integrated with Cross-Origin Resource Sharing (CORS).
In contrast, web sockets are raw TCP sockets that share nothing with HTTP except for the ability for a user agent to use the HTTP protocol to initiate a web socket connection. After that, everything is up to agreements between the client and server.
Since yada is designed to support HTTP, it does not provide anything extra to support web sockets beyond that which is provided by the web server.
17.2. SSE with yada
To create Server Sent Event streams with yada, return a stream of data from a response.
For example, a stream of data could be a core.async channel. It is
important that you set the representation to be text/event-stream
, so
that a client recognises this as a Server Sent Event stream and keeps
the connection open.
(require '[clojure.core.async :refer [chan]])
{:methods {:get {:produces "text/event-stream"
:response (chan)}}}
It is, however, highly unusual to want to provide a channel of data to a
single client. Typically, what is required is that each client gets a
copy of every message in the channel. This can be achieved easily by
multiplexing the channel with clojure.core.async/mult
, which yada will
recognise and tap on your behalf.
(require '[clojure.core.async :refer [mult]])
(let [mlt (mult channel)]
{:methods {:get {:produces "text/event-stream"
:response mlt}}})
Of course, you can tap
the mult
yourself in your own logic and
provide the tapping channel directly to yada, which will do the right
thing depending on what you provide.
22. The Request Context
When given the HTTP request, the handler first creates a request-context and populates it with various values, such as the request and the resource-model that corresponds to the request’s URI.
The handler then threads the request-context through a chain of functions, called the interceptor-chain. This chain is just a list of functions specified in the resource-model that has been carefully crafted to generate a response that complies fully with HTTP standards. However, as with anything in the resource-model, you can choose to modify it to your exact requirements if necessary.
The functions making up the interceptor-chain are not necessarily executed in a single thread but rather an asynchronous event-driven implementation enabled by a third-party library called manifold.
23. Interceptors
The interceptor chain, established on the creation of a resource. A resource’s interceptor chain can be modified from the defaults.
23.2. Modifying interceptor chains
Say you want to modify the interceptor chain for a given resource.
You might want to put your interceptor(s) at the front.
(yada.resource/prepend-interceptor
resource
my-interceptor-a
my-interceptor-b)
Alternatively, you might want to replace some existing core interceptors:
(update resource
:interceptor-chain
(partial replace {yada.interceptors/logging my-logging
yada.security/authorize my-authorize}))
Or you may want to insert some of your own before a given interceptor:
(yada.resource/insert-interceptor
resource yada.security/authorize my-pre-authorize)
You can also append interceptors after a given interceptor:
(yada.resource/append-interceptor
resource yada.security/authorize my-post-authorize)
24. Sub-resources
Usually, it is better to declare as much as possible about a resource prior to directing requests to it. If you do this, your resources will expose more information and there will be more opportunities to utilize this information in various ways (perhaps serendipitous ones). But sometimes it just isn’t possible to know everything about a resource, up front, prior to the request.
A classic example is serving a changing directory of files. Each file might be a separate resource, identified by a unique URI and have different possible representations. Unless the directory’s contents are immutable, you should only determine the number and nature of files contained therein upon the arrival of a request. For this reason, yada has sub-resources. Sub-resources are resources that are created just-in-time when a request arrives.
24.1. Declaring sub-resources
Any resource can declare that it manages sub-resources by declaring a :sub-resource_ entry in its resource-model. The value of a sub-resource is a single-arity function, taking the request-context, that returns a new resource, from which a temporary handler is constructed to serve just the incoming request.
Sub-resources are recursive. A resource that is returned from a sub-resource function can itself declare that it provides its own sub-resources, ad-infinitum.
24.2. Path info
When routing, it is common for resources that provide sub-resources to
match a set of URIs all starting with a common prefix, and extract the
rest of the path from the request’s :path-info
entry. This is achieved
by declaring a :path-info? entry in the resource-model set to true.
(resource
{:path-info? true
:sub-resource
(fn [ctx]
(let [path-info (get-in ctx [:request :path-info])]
(resource {…})))})
(For a good example of sub-resources, readers are encouraged to examine
the code for yada.resources.file-resource
to see how yada serves the
contents of directories.)
26. Testing
Using yada.response-for
, you can create a response from a bidi route
structure containing yada resources for testing purposes:
(response-for ["/foo" (yada "hello")] :get "/foo")
Reference
Appendix A: Reference
A.4. Protocols
yada defines a number of protocols. Existing Clojure types and records can be extended with these protocols to adapt them to use with yada.
A.4.2. yada.methods.Method
Every HTTP method is implemented as a type which extends the yada.methods.Method protocols. This way, new HTTP methods can be added. Each type must implement the correct semantics for the method, although yada comes with a number of built-in methods for each of the most common HTTP methods.
Many method types define their own protocols so that resources can also
help determine the behaviour. For example, the built-in GetMethod
type
uses a Get
protocol to communicate with resources. The exact semantics
of this additional protocol depend on the HTTP method semantics being
implemented. In the Get
example, the resource type is asked to return
its state, from which the representation for the response is produced.
A.4.3. yada.body.MessageBody
Message bodies are formed from data provided by the resource, according to the representation being requested (or having been negotiated). This removes a lot of the formatting responsibility from the resources, and this facility can be extended via this protocol for new message body types.
A.5. Built-in types
There are numerous types already built into yada, but you can also add your own. You can also add your own custom methods.
A.5.1. Files
The yada.resources.file-resource.FileResource
exposes a single file in
the file-system.
The record has a number of fields.
Field
Type
Required?
Description
file
java.io.File
yes
The file in the file-system
reader
Map
no
A file reader function that takes the file and selected representation, returning the body
representations
A collection of maps
no
The available representation combinations
The purpose of specifying the reader
field is to apply a
transformation to a file’s content prior to forming the message payload.
For instance, you might decide to transform a file of markdown text content into HTML. The reader function takes two arguments: the file and the selected representation containing the media-type.
(fn [file rep]
(if (= (-> rep :media-type :name) "text/html")
(-> file slurp markdown->html)
;; Return unprocessed
file))
The reader function can return anything that can be normally returned in the body payload, including strings, files, maps and sequences.
The representation
field indicates the types of representation the
file can support. Unless you are specifying a custom reader
function,
there will usually only be one such representation. If this field isn’t
specified, the file suffix is used to guess the available representation
metadata. For example, a file with a .png
suffix will be assumed to
have a media-type of image/png
.
A.5.2. Directories
The yada.resources.file-resource.DirectoryResource
record exposes a
directory in a filesystem as a collection of read-only web resources.
The record has a number of fields.
Field
Type
Required?
Description
dir
java.io.File
yes
The directory to serve
custom-suffices
Map
no
A map relating file suffices to field values of the corresponding FileResource
index-files
Vector of Strings
no
A vector of strings considered to be suitable to represent the index
A directory resource not only represents the directory on the file-system, but each file resource underneath it.
The custom-suffices
field allows you to specify fields for the
FileResource records serving files in the directory, on the basis of the
file suffix.
For example, files ending in .md
may be served with a FileResource
with a reader that can convert the file content to another format, such
as text/html
.
(yada.resources.file-resource/map->DirectoryResource
{:dir (clojure.java.io "talks")
:custom-suffices {"md" {:representations [{:media-type "text/html"}]
:reader markdown-reader}}})
Colophon
This book is authored in AsciiDoc, which is a plain-text format like Markdown. AsciiDoc has the benefit of being based on the mature DocBook standard.
Sources for this book can be found in the yada repository under the doc/
directory. The file book.adoc
describes the structure of the book.
HTML
Asciidoctorj is used to generate HTML.
The main font is Noto Sans, licensed under the Open Font License.
Monospace
font is Droid Sans Mono, licensed under the Apache License.
The yada font is based on Rochester licensed under the Apache License, Version 2.0.
Asciidoctor is used to generate DocBook 5 XML which is output to LaTeX using dblatex.