A RESTless week

published on February 16, 2016

I spent about a week of my time at Acquia on improving Drupal 8’s REST support.

That time was spent fixing, reviewing, triaging and documenting.

Drupal 8’s REST works very well, we just have to make it more friendly & helpful, remove Drupalisms and support more REST capabilities.

Fixing, reviewing & triaging

I went through the entire issue queues of rest.module, serialization.module and hal.module. I was able to mark about a dozen bug reports as duplicates, fix a dozen or so support requests, have reviewed probably a few dozen patches, rerolled about a dozen patches, created at least another dozen patches and … triaged 100% of the open issues. I clarified titles of >30 issues.

Now the rest.module issue queue (the most important one) fits on a single page again!1 I collaborated a lot with neclimdul, klausi, damiankloip, dawehner and others.

dawehner and I decided to tag the issues that were especially relevant using the RX (REST Experience) issue tag.

I felt it was important to get a comprehensive picture of Drupal 8’s REST support state, so I insisted on going through all open issues (and was given the time to do so). This enabled me to document the current state of things (and upcoming improvements).

Documenting

So, I spent several days doing nothing else but writing and improving documentation, just like I did for the modules and subsystems I co-maintain. The following drupal.org handbook pages have either received minor updates, received complete overhauls or were written from scratch:

So, there you go, documentation covering fundamentals, like that for the Serialization API:

Serializing & deserializing
Using the serializer service’s (\Symfony\Component\Serializer\SerializerInterface) serialize() and deserialize() methods:
$output = $serializer->serialize($entity, 'json');
$entity = $serializer->deserialize($output, \Drupal\node\Entity\Node::class, 'json');
Serialization format encoding/decoding (format → array → format
The encoder (\Symfony\Component\Serializer\Encoder\EncoderInterface) and decoder (\Symfony\Component\Serializer\Encoder\DecoderInterface, to add support for encoding to new serialization formats (i.e. for reading data) and decoding from them (i.e. for writing data).
Normalization (array → object → array)
The normalizer (\Symfony\Component\Serializer\Normalizer\NormalizerInterface) and denormalizer (\Symfony\Component\Serializer\Normalizer\DenormalizerInterface), to add support for normalizing to a new normalization format. The default format is as close to a 1:1 mapping of the object data as possible, but other formats may want to omit e.g. local IDs (for example node IDs are local, UUIDs are global) or add additional metadata (such as URIs linking to related data).
Entity resolvers
In a Drupal context, usually it will be (content) entities that end up being serialized. When given an entity to normalize (object → array) and then encode (array → format), that entity may have references to other entities. Those references may use either UUIDs (\Drupal\serialization\EntityResolver\UuidResolver) or local IDs (\Drupal\serialization\EntityResolver\TargetIdResolver). For advanced use cases, additional mechanisms for referring to other entities may exist; in that case, you would add an additional entity resolver.

… to more practical information, such as the Getting started: REST configuration & REST request fundamentals handbook page for the rest module:

1. Configuration

First read RESTful Web Services API — Practical.

Now you know how to:

  1. expose data as REST resources
  2. grant the necessary permissions
  3. customize a REST resource’s formats (JSON, XML, HAL+JSON, CSV …)
  4. customize a REST resource’s authentication mechanisms (cookie, OAuth, OAuth 2.0 Token Bearer, HTTP Basic Authentication …)

Armed with that knowledge, you can configure a Drupal 8 site to expose data to precisely match your needs.

2. REST request fundamentals

2.1 Safe vs. unsafe methods

REST uses HTTP, and uses the HTTP verbs. The HTTP verbs (also called request methods) are: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT and PATCH.
Some of these methods are safe: they are read-only. Hence they can never cause harm to the stored data, because they can’t manipulate them. The safe methods are HEAD, GET, OPTIONS and TRACE.
All other methods are unsafe, because they perform writes, and can hence manipulate stored data.

Note: PUT is not supported for good reasons.

2.2 Unsafe methods & CSRF protection: X-CSRF-Token request header

Drupal 8 protects its REST resources from CSRF attacks by requiring a X-CSRF-Token request header to be sent when using a non-safe method. So, when performing non-read-only requests, that token is required.
Such a token can be retrieved at /rest/session/token.

2.3 Format

When performing REST requests, you must inform Drupal about the serialization format you are using (even if only one is supported for a given REST resource). So:

  1. Always specify the ?_format query argument, e.g. http://example.com/node/1?_format=json.
  2. When sending a request body containing data in that format, specify the Content-Type request header. This is the case for POST and PATCH.

Note: Accept-header based content negotiation was removed from Drupal 8 because browsers and proxies had poor support for it.

3. Next

Now you’re ready to look at concrete examples, which start on the next page.

If that particular handbook page had already existed, it would have saved me so much time! The next page then contains examples for how to do GET requests, using various tools:

cURL

curl http://example.com/node/1?_format=hal_json

Guzzle

$response = \Drupal::httpClient()
  ->get('http://example.com/node/1?_format=hal_json', [
    'auth' => ['username', 'password'],
  ]);

$json_string = (string) $response->getBody();

jQuery

jQuery.ajax({
  url: 'http://example.com/node/1?_format=hal_json',
  method: 'GET',
  success: function (comment) {
    console.log(comment);
  }
});

… and the following pages then provide concrete examples (in those same tools) for POST, PATCH and DELETE requests2.

Enjoy!

Why I did all of the above

It took me about three days to successfully PATCH a Comment entity3.

Why days?

I first forgot to specify the Content-Type request header. Then it turned out I also forgot the X-CSRF-Token request header — which was not documented anywhere to be a thing. I eventually found out about that Drupal-specific request header by analyzing the REST PATCH test coverage. Why did I not find it sooner? Because Drupal 8 was giving utterly unhelpful, and actually downright nonsensical (and incorrect!) responses4. It doesn’t end there though. Turns out that if you try to update an entity using JSON (and not HAL+JSON, which works fine), you MUST specify the bundle (otherwise it’s impossible to denormalize the entity you’re sending), but you also MUST NOT specify the entity type’s bundle if it’s a Comment (because you’re not allowed to modify this by CommentAccessControlHandler). So … it literally was impossible to update a comment5!

I didn’t have any experience with/knowledge about Drupal 8’s REST API. But I’m deeply familiar with Drupal 8. And it still took me days. Of course I wanted to prevent anyone from ever having to go through that.

Today, anybody would start at d.o/documentation/modules/rest/start and then look at d.o/documentation/modules/rest/patch and would hence be able to avoid all these pitfalls. Soon, Drupal will provide more helpful responses4 and allow comments to be updated5 using JSON.


  1. Once the “fixed” issues disappear. 50 issues fit on a single page. I didn’t count the number of issues before I started, but it was at least 70, and I think ~90. 

  2. These handbook pages already existed mostly, but lacked clarity, coherence and completeness. That’s what I tried to add. 

  3. We’re working on an experiment in progressive decoupling. For that to work, you of course need solid REST support. Once those experiments become worth sharing, we will. 

  4. I’m fixing that in https://www.drupal.org/node/2659070

  5. I’m fixing that in https://www.drupal.org/node/2631774

Comments

Sven Decabooter's picture
Sven Decabooter

Thanks a lot for all the documentation updates, Wim. Much appreciated!

Ryan's picture

off topic - what are you using for your footnotes?

Kay V's picture

“I wanted to prevent anyone from ever having to go through that” - and I can attest that your efforts paid off. Nice work! It is very evident in the docs, as well!

Excellent job writing clear, concise and dead straight-forward background, explanations and instructions. No small feat!

Jelle De Loecker's picture

I (unfortunately) did not find your blog post in time.

I do not know Drupal very well, and I had to create a custom RestResource in it. Every problem you had, I had. In the same order.

The Content-Type and CSRF token being required without being documented anywhere (I finally found out about it in some tutorial)

And I’m still struggling with the final problem: posting regular JSON to my custom POST resource.

All I get is Could not denormalize object of type , no supporting normalizer found, even though I don’t want it to normalize anything. It’s just a damn object.

Kay V's picture

Relationships between entities (one of Drupal’s outstanding distinctions) seem particularly fraught in POSTs even though GETs in my experience have worked flawlessly. Happy to help expand documentation on d.o. assuming I can find a way forward. Any recommendations on cracking this piece of the puzzle?

Wim Leers's picture
Wim Leers

When using the default normalization and JSON encoding, that hasn’t been tricky at all. I suspect you’re referring to the application/hal+json format?

swentel's picture
swentel

This is very valuable information, thanks! Do we have some documentation somewhere (or tips) on the best approach to upgrade from say D6 to D8 with existing services (from the services module). The problem is that there’s a lot of external applications (iOS, Android, Other) consuming that API, so existing paths/routes should be the same when we launch the D8 version of the site. However, with _format=json argument and the X-CSRF-Token for unsafe requests, this might become a little tricky. Although I could probably just swap everything out anyway as I’ll have to define my own resources to match the existing requests, nothing much I can use for which is core. Sorry for the late night rambling :)

Wim Leers's picture
Wim Leers

I don’t know of any such documentation. Would be a good blog post or — better yet — a good Drupal.org handbook page :)