Improving Drupal's page loading performance

Introduction

Google dominates the search engine market for a large part thanks to its spartan, no-bells-nor-whistles interfaces. But also thanks to its incredible speed (which is partially thanks to that spartan interface, of course).

Since you’re reading this article, you’re probably a Drupal developer. It’s pretty likely that you’ve had some visitors of your Drupal-powered web site complain about slow page load times. It doesn’t matter whether your server(s) are shared, VPSes or even dedicated servers. Visitors that live abroad – i.e. far from where your servers are located – will face the same performance issues, but at even worse scales.
This article is about tackling these issues.

Front-end performance

Faster servers with more memory stop improving your web site’s performance at some point. Yet, even before your web site gets big, there are other places to look at to improve performance, where greater effects can be achieved, even at lower costs – significantly lower costs actually. Typically, less than 20% of the total response time is used to retrieve the HTMl document. That means the other 80+% is used to process what’s in the HTML file: CSS, JS, images, videos. And in many cases, that number is even higher.

Depending on your website, your server(s), et cetera, these optimizations will probably shave off between 25 and >100 percent (estimated) of your page loading time. Initial (empty cache) and consecutive page loads (primed cache) will both be significantly faster, unless you’ve already done your own round of optimizations.

Much thanks go to Yahoo!’s research that resulted in fourteen rules and the accompanying YSlow tool (we’ll get to that in a second) that allows you to check how your web site performs according to those rules. If you can apply all fourteen successfully, your web site should fly. (Assuming that your page generation times aren’t super slow, of course.) As always, more optimizations are still possible. I’ll discuss some very effective ones briefly at the end.

YSlow

First things first: make sure you’ve installed Firefox, Firebug and YSlow for Firebug (version 0.9 or better).

Firebug is simply a must-have for any web developer, it doesn’t matter whether you’re a professional or an amateur. YSlow is a Firefox add-on developed by Yahoo!, that analyzes your web page and tells you why exactly (remember those fourteen rules?) your site is slow (hence “y-slow”, which is pronounced as “why slow”). But at the same time, it tells you how you can fix those pain-points. The lower the rule number, the greater the effect.

What follows is a comprehensive, yet pretty complete review of how Drupal 5 and 6 score on each rule, by listing the required features, settings or guidelines.

If you want to skip the information and want to see results, just skip to the part where I explain how you can apply the optimizations to your site.

Rule 1: Make fewer HTTP requests

Requirement	Drupal 5	Drupal 6
CSS aggregation	yes	yes
JS aggregation	no	yes
Generate CSS sprites automatically	no	no

Drupal even has the ability to compress CSS files (through stripping comments and whitespace). JS aggregation has been added in Drupal 6. To my knowledge, not a single CMS/CMF ships with the ability to generate CSS sprites. Nor does a single one have a module or extension that allows them to do so. This could be a Drupal key performance feature, if it were supported.

Solution

The easiest way to reduce this significantly is to enable Drupal’s CSS and JS aggregation. You can find these settings at admin/settings/performance in your Drupal site.
If you’re using Drupal 5, there’s a backport of Drupal 6’s JS aggregation feature, you can find it in this issue – I sponsored this patch.

There is not yet an automatic CSS sprite generator module for Drupal. If your site is styled pretty heavily, this would benefit you even more than CSS and JS aggregation. I hope somebody – or some Drupal company – will take the initiative.
In the mean time, there’s a free CSS Sprite Generator out there, if you don’t mind doing it manually.

Rule 2: Use a CDN

Requirement	Drupal 5	Drupal 6
Alter URLs of served files dynamically	no	no

Drupal’s File API needs work: it should be trivial to alter file URLs dynamically, e.g. based on the file size or type of a file.

Solution

I chose to tackle this particular problem myself, because using a CDN greatly enhances the usability of your web site for visitors that live far away from your servers. And one of the projects I’m working on, is one with a very international audience.

The first part of what’s needed, is obviously to update Drupal core to support file URL altering. I chose to create a new function, file_url(), through which all URLs for files should be generated, including the URLs for additional CSS files in the page.tpl.php file (e.g. for a print.css file). This patch also provides a new hook: hook_file_server(), through which modules can provide new file servers. To configure the preferred file server, a new “File servers” setting has been added to the File system settings form. If one server can’t serve a file, Drupal will try the second server, and so on. It will always fall back to the web server Drupal is being served from if all servers provided by modules failed.
Currently, I’ve only got a Drupal 5 patch (it’s included in the CDN integration module and attached at the bottom of this article), because I want to get more feedback before I start maintaining patches for 2 different versions of Drupal. As soon as the patch ends up in its final form, I will provide a Drupal 6 patch, and of course push for Drupal 7 inclusion. An issue at Drupal.org has been created.

The second part – integration with a CDN – obviously requires an implementation of hook_file_server(). So the CDN integration module was born. It’s written with flexibility in mind: it supports synchronization plugins (currently ships with one: FTP), can create unique filenames or directories (necessary if you don’t want to break relative paths), provides the tools to check whether your filters are working well (per-page and site-wide statistics) and the filters can be configured using parameters similar to Drupal’s file_scan_directory() function.

An article that includes benchmarks of the effects of the CDN integration module is being worked on. The same article will include a complete installation tutorial as well.

Rule 3: Add an Expires header

Requirement	Drupal 5	Drupal 6
Don’t set the Expires header for web pages	yes	yes
Set the Expires header for all other files	yes	yes
Allow far future Expires headers: ability to alter URLs of served files dynamically	no	no

By setting the Expires header for files, you tell the browser that it’s ok to cache them.
Drupal sets the “Expires” header for all other files than web pages to 2 weeks. This is sufficient for most uses. For maximum performance, this should be set to a date in the far future (e.g. 10 years from access), but this requires unique filenames: each time the file is updated, the filename should change as well this is why file URL altering is a requirement. If not, your users could still be using the old files, since they may be in their cache.

Solution

Changing the future date for the Expires headers is easy enough: simply edit your .htaccess file. Your Apache server must also have mod_expires installed, this is available by default on most servers. However, making filenames unique is an entirely other matter. The altering of file URLs is already solved in the solution for rule 2. So all you have to do now, is implementing a file server that supports this. The aforementioned CDN integration module provides this feature, but if you want to use it, you of course have to use a CDN.

Rule 4: GZIP components

Requirement	Drupal 5	Drupal 6
GZIP web pages	yes	yes
GZIP CSS and JS files	no	no

When Drupal’s page caching is enabled, pages are written to the cache in GZIPped form! To learn more about how Drupal handles GZIPping, run this command from your Drupal root directory:

egrep ‑rn "gzip" .

Don’t forget the dot at the end!
However, Drupal does not yet allow you to gzip CSS and JS files.

Solution

A Drupal core patch for this is being worked on, but has unfortunately been inactive for quite some time.
If you are using my CDN integration module, you don’t need to worry about this, since CDNs GZIP files by default, if the client supports it.

Alternative solution

As an alternative, you could configure your Apache server to automatically compress files.
An example for Apache 2.x: add the following lines to your .htaccess or httpd.conf file:

AddOutputFilterByType DEFLATE text/css application/x-javascript

Rule 5: Put CSS at the top

Requirement	Drupal 5	Drupal 6
Abstraction to add CSS files to the web page	yes	yes
Default location in the XHTML document is the tag	yes	yes

Drupal has this abstraction: drupal_add_css().
Putting stylesheets to the document HEAD makes pages load faster: it allows the page to be rendered progressively.

Rule 6: Put JS at the bottom

Requirement	Drupal 5	Drupal 6
Abstraction to add JS files to the document	yes	yes
Default location in the XHTML document is just before	no	no

Drupal has this abstraction as well: drupal_add_js().
JS should be at the bottom, because browsers wait until everything in the tag has loaded. As you probably know, JS files tend to be pretty large these days, so loading them might take a while, thus postponing the rendering of the page. If you’d put the JS files at the bottom, then your page can be rendered while the JS files are still loading! It also achieves a greater download parallelization, thus cutting down your overall page loading time.
This is also being discussed at groups.drupal.org.

Solution

Unfortunately, the default value for the $scope parameter of drupal_add_js() is bad: 'header'. If we simply make 'footer' the default, we’re good. The number of contributed modules that sets this to 'header' explicitly, is very low, so it shouldn’t be too much work to convert these. And I’ve yet to encounter the first module that has issues with being at the bottom instead of the top.

A more complex part of the solution are Drupal’s default JS files: misc/jquery.js and misc/drupal.js. These can be put in the footer without any issues whatsoever. But what if a contributed module chooses to put its files in the header? Then they may not yet be loaded! For maximum compatibility, we should add the default JS files to the header if at least one module chooses to add its JS file to the header.

I’ve attached patches for both Drupal 5 and 6, but neither implement the more complex part I just explained. In my opinion, Drupal should enforce a strict policy: all JS files should be “footer-compatible”. Until somebody can point me to some JS that must be in the header to work properly, I’m unlikely to change my opinion about this proposed policy.

Alternative solution

The second method to fix this, doesn’t involve hacking Drupal core, but is also more hassle since you have to repeat it for every theme you’re using. Suppose you’re using the default Drupal core theme, Garland. Then open the themes/garland/page.tpl.php file in your favorite editor. Find this line at the top of the file:

Cut it away from there, and put it just before this line at the bottom:

So your end result should look like this:

As you can see, it now comes just before the closing tag. (Well, also before the $closure output, which is the output generated by all hook_footer() implementations.)

Rule 7: Avoid CSS expressions

Requirement	Drupal 5	Drupal 6
No default theme should implement it	yes	yes

CSS expressions are not recommended, because they’re evaluated many times: when the page is rendered or resized, but also when the page is scrolled. Even when the user moves the mouse over the page!
None of the Drupal core themes uses CSS expressions. Just make sure you don’t use it in your own themes.

Rule 8: Make JS and CSS external

Requirement	Drupal 5	Drupal 6
Inline CSS and JS should be avoided or used sparingly	yes	yes

If your web site is a common homepage for many users, you may want to use a custom strategy and read this. Otherwise, you can ignore this rule without worrying.

Rule 9: Reduce DNS lookups

Requirement	Drupal 5	Drupal 6
Use 2-4 hostnames by default: ability to alter URLs of served files dynamically	no	no

To my knowledge, not a single hosting provider offers a static file server by default. So it makes a lot of sense that Drupal doesn’t do it by default. However, Drupal should support it out-of-the-box.

If you use a lot of so called widgets (those small blocks of content provided by Flickr, del.icio.us, MyBlogLog, and so on) on your site, you’ll have some extra work to do.

Solution

The altering of file URLs is already solved in the solution for rule 2. So, once again, all you have to do now, is implementing a file server that supports this.
If you use my CDN integration module, then you’ll be using 2 hostnames or more, but this of course requires you have access to a CDN.
Alternatively, you can use a static file server. Robert Douglass’ article on using Drupal with a static file server is a very complete reference: from the pros and cons to the entire server setup.
See Yahoo’s documentation for more details.

Widgets solution

If you’re using a lot of widgets and you want to continue using them, you can. The solution is simple: cache as much as possible on your own site (or to your CDN).
For example, if you use Google Analytics, make sure you’ve installed the Google Analytics module, which has an option to cache the .js file locally (and update it once each day, to make sure you’re serving the latest version).

Rule 10: Minify JavaScript

Requirement	Drupal 5	Drupal 6
JavaScript minification	no	no

This was originally included in Drupal 6 core. However, it has been removed because it was problematic; it would result in boatloads of JS errors, and thus the JS would simply stop working. The technique used by Drupal 6 was Dean Edwards’ packer, also nicknamed packer.

Solution

Packer, isn’t just a minifier, it’s also an obfuscator. A minifier only strips whitespace and comments, but an obfuscator also munges the code; it renames variable and functions as short as possible. For this, the packer uses a reduction algorithm (hence its name). However, this has serious consequences for the page loading time as well: it can easily take 200 ms to unpack the JS! Additionally, the effectiveness of GZIPping packed JS files is much lower.

More reliable alternatives are JSMIN (minifier/uglifier), Dojo Shrinksafe (minifier/obfuscator) and YUI Compressor (minifier). The last two are built on top of Rhino, Mozilla’s Javascript engine, written in Java. Therefor neither qualify for Drupal core inclusion. A JSMIN PHP implementation exists, so I think this is the best choice.
There’s an issue to add this to Drupal 7.

Rule 11: Avoid Redirects

Requirement	Drupal 5	Drupal 6
Avoid redirects by default	yes	yes

Drupal could redirect users accessing the non-aliased URL /node/11 to the aliased version /about, but it does not – at least not by default.
The Global Redirect module implements this feature in a sensible way. See the project page for a complete explanation.

Rule 12: Remove Duplicate Scripts

Requirement	Drupal 5	Drupal 6
Abstraction to add JS files to the document	yes	yes

Drupal has this abstraction, as mentioned in rule 6: drupal_add_js(). You then just use a static variable to prevent adding the same file multiple times. For an example, see the jCarousel module.

Rule 13: Configure ETags

This is the only rule that depends completely on the server setup. An ETag uniquely identifies a file, and is used to verify if the file in the browser’s cache matches the file on the server.

The problem is that they are generated using attributes specific to the server (inodes) they’re being served from. This implies that when, for example, you’re using multiple servers behind a load balancer, you may one time be accessing the files from server 1, another time the files from server 2. And since the ETags don’t match, you’ll be downloading the file again!

Solution

If you’re using multiple servers, disable ETags. For Apache users: add this line to your httpd.conf:

FileETag none

IIS users have to follow a more complex procedure.

Alternative solution

Use a single server to serve files, or a CDN. See the solutions for rules two and nine for detailed information.

Rule 14: Make Ajax Cacheable

Requirement	Drupal 5	Drupal 6
Pluggable render formats	no	no
Ability to set GZIP-ability per format (i.e. rule 4)	no	no

These feature was being worked on for Drupal 6, but unfortunately wasn’t ready in time.
The ability to set the GZIP-ability per output format (i.e. rule 4) should be handled at in the same patch, or in a follow-up, since it affects the performance of AJAX callbacks so much.
Other - but mostly less important - rules that are implemented automatically: rules 9 and 13 (AJAX responses will be served from the same server as Drupal), rule 11 (redirects are extremely unlikely to be used for AJAX callbacks in Drupal). Rule 10 is almost completely irrelevant, because GZIPping JSON data has a much greater effect.

Solution

The node rendering refactoring issue is listed for the Drupalcon Boston 2008 code sprint, so we’ll probably see this in its final form in a couple of months.

Applying this to your site

This is of course a boatload of information. The easiest way to apply all of it to your site, is by installing my CDN integration module, and using the included Drupal core patch that also adds JS aggregation and puts JS files at the bottom (by changing the default scopes).

Live sites

Before you start applying this to your site, you of course want some proof that all these optimizations do indeed have an effect. No problem. You’re looking at it. This page should have been loaded in less than a second. Subsequent page loads should complete in less than half a second. With Drupal’s page caching disabled (!), eAccelerator installed and a MySQL query cache in place.

Another live site is DrupalBin. That site is running on a shared server (DreamHost), without eAccelerator and without a MySQL query cache – which explains the often slow page generation times.

Additional optimizations

In order of effectiveness:

• The Boost module enables Drupal to perform static page caching. This means that rendered pages are written to files, and through some mod_rewrite magic, it will serve the statically cached page from the file if it’s available, thus without even a single DB query!
• This article at 2bits is chock full with Drupal performance tips.
• The core patches module Advanced cache, again by Robert Douglass, provides caching for blocks, comments, the forum structure, built nodes, path lookups, popular search queries and taxonomy trees.

Drupal issues

• #101227: GZIP aggregated CSS and JS, by Owen Barton (Grugnog2) and Ted Serbinski (m3avrck).Has some patches, but is out of sync.
• #210447: Introduce Javascript compression, by Nedjo Rogers (nedjo). Doesn’t have patches yet.
• #134478: Refactor node rendering, by Jeff Eaton (eaton) et al.
• #214934: file_url() and hook_file_server(), by Wim Leers.

More information

• High Performance Web Sites (video of presentation), by Steve Souders.
  Explains all YSlow rules in detail.
• CSS Sprites: Image Slicing’s Kiss of Death, by Dave Shea, A List Apart.
  Explains the need for CSS sprites, how they work and the pittfalls.
• CSS Sprites: How Yahoo.com and AOL.com Improve Web Performance, websiteoptimization.com.
  Case-study that may be interesting if you’re learning about CSS sprites.
• CSS Sprite Generator Tool, by Stuart Colville, Muffin Research Labs.
  Announcement of the aforementioned CSS sprite generator.
• Announcing SilkSprite: A CSS Sprite Plugin for Blueprint, by Don Albrecht, Ajax Bestiary.com.
  Automatic CSS sprite generator for the Blueprint CSS framework.
• Performance Research, Part 2: Browser Cache Usage - Exposed!, by Tenni Theurer’s, Yahoo! User Interface Blog.
  Aptly describes why it’s important to not forget about page views with an empty cache.
• Serving JavaScript Fast, Cal Henderson, Vitamin.
  A pretty complete reference on how to serve GZIPped CSS and JS files.
• Gzip Your Minified JavaScript Files, Julien Lecomte.
  A comparison of how well GZIP-able minified JS files are, after using a certain minifier. Also explains why the packer should not be used due to its reduction algorithm.
• Static Page Caching for Drupal 4.7, Arto Bendiken.
  Announcement of the Boost module.
• Page Load Time Worksheet, Tag 1 Consulting, Inc.
  This worksheet can help you to isolate and fix bottlenecks affecting your page load times.

Thanks to …

• Yahoo! for their work on YSlow.
• Greg Knaddison (greggles) for proofreading this article and making several excellent suggestions to make this article more complete.