Episodes: Drupal integration & ingestor

Published on 15 March, 2009

In my session at DrupalCon DC, I promised an initial version of the Episodes module by March 15, which is today. I’m glad to be able to announce that I somewhat met that goal.

If you don’t know what it is exactly, I encourage you to read the project description first.

Status {#status}

It’s not yet completely finished: the basic reporting UI must still be written. But you can already look at the results of each individual page through the Firebug add-on (which I didn’t write, it’s already available). See the first screenshot for that. That’s of course much less useful, but it gives you a clear indication of the potential.
However, before I do that, I first have to work on making other deadlines for other courses.
So what’s done already? Here’s an overview:

  • Setting to enable or disable the Episodes integration or put it in debug mode.
  • Setting to disable or enable logging.
  • Setting to not apply the Episodes integration on certain Drupal paths. Wilcards can be used.
  • The functionality to scan all .js files in the Drupal installation and filter out all Drupal.behaviors (this contains all JavaScript behaviors of the current web page). Uses Drupal’s batch API and is therefore scalable. Stores the results in the database and provides a simple UI (based on Hierarchical Select) to allow you to ignore some Drupal.behaviors, for example those that never have a significant impact on page loading performance. Look at screenshot number two to see what this looks like.
  • Tweaked Steve Souders’ sample episodes.js a lot:
    • Ended all functions with semi-colons to prevent JS errors.
    • Used EPISODES.compatScriptUrl instead of a hardcoded URL.
    • Fixed typo: renamed startMakeName variable to startMarkName.
    • Fixed the referrer matching logic in EPISODES.findStartTime().
    • Extended EPISODES.sendBeacon() to actually send the beacon using an AJAX GET request.
    • Extended EPISODES.init() to include pageready marks and backend, pageready, frontend episodes.
  • Created Drupal.Episodes (this is JavaScript), which provides automatic integration with Drupal. Each behavior in Drupal.behaviors is automatically turned to in its own episode! Other features:
    • Measures the domready episode.
    • Has the ability to chain lazy load callbacks and measure these as well.
    • Measures the totaltime episode in a smart way: if any lazy load callbacks are defined, it’s done relative to those, otherwise it’s measured relative to the pageready mark.
  • Ensured compatibility with all of Drupal’s page caching modes, including aggressive page caching.
  • Measuring the episodes.js, css, headerjs and footerjs episodes is possible, but impossible without changing your theme’s page.tpl.php. Clear instructions are included in the README.
  • Figured out a solid httpd.conf setup, that will also work for sites installed in subdirectories.
    • Wrote an ingestor that moves all collected data from the Apache log file into the database. The same database as the one for Drupal is used for now, but this can change in the future. In the end, you really want the logging server to be separate from your application server, at least in a separate VM with limited resources. It also extracts the browser name, browser version and operating system reliably thanks to Browser.php (see below).

Libraries used or looked at for inspiration:

All of this has been written in less than 4 days (revisions 5-15). I’d expect somebody who isn’t familiar with Drupal to need at least 3 times that amount of time and if you’re not familiar with JavaScript another couple of days. But it of course also took a substantial amount of research (reading) about how all these things connect with each other: preparation is key to productivity!

Try it! {#try-it}

Want to give it a try? Perfectly possible, but it’s not yet committed to cvs.drupal.org, and I think I’ll keep it that way. It’s easier for me to keep all my work in a single SVN repository (which is also faster and nicer to work with than CVS). Whenever this module is its final shape, I’ll commit it to cvs.drupal.org — the same will apply to the CDN integration module that I’ll be rewriting as part of this thesis.

So, for now, get the code from my personal repository (link removed). But please do use the issue queue at Drupal.org for reporting bugs and requesting features. The Episodes project is now available at drupal.org.

  • 27 views