Wim Leers - Drupal

Bachelor thesis on Drupal's page loading performance

Wim Leers — Fri, 17 Oct 2008 23:16:22 +0000

I've alluded to it before, but now it's also been officially approved: I'll be doing my bachelor thesis on Drupal! I will focus on integrating Drupal with CDNs. Yay! :)

Don't know what a CDN is? It's short for Content Delivery Network; a network of (static file or streaming media) servers that are located around the globe. These servers all mirror each others' files. When a user requests a certain file from the CDN, the server that is the closest to the user will serve the file.
By using a CDN to serve the static components on your web site (CSS, JS, images, fonts), your web site will load much faster: the latency will be lower and the throughput will be greater.

You'd think that the tools necessary to synchronize ﬁles to a CDN are already available. Unfortunately, that’s not the case. The most important reason is that CDNs have only become cheaper very recently. Most CDNs assume that they are going to be used for the distribution of big ﬁles, and therefore a manual upload – typically via (S)FTP – of each ﬁle is acceptable. It's not acceptable for dynamic (Drupal) web sites though!

I will take the CDN integration module that I wrote, extract the useful pieces and then rewrite it properly (i.e. actually scalable) for Drupal 6. Not only is the goal to write a scalable Drupal module (i.e. synchronization through PHP), but also to create a daemon. This daemon (in C++/Qt) will allow for far more efficient synchronization, because you don't have to scan directories recursively for new and changed files anymore, you can take advantage of file system event monitors (inotify on Linux, FSEvents on Mac OS X and WMI on Windows) to track files without creating overhead.

Interested? For all details (the above really is just an extract), see my proposal (English, 5 pages) at the bottom of this blog post.

How much is too much?

Wim Leers — Tue, 14 Oct 2008 17:18:07 +0000

So, university has started again. We have to pick "broadening courses", I chose economy courses (macroeconomics and operational research). In this semester, I've got 6 courses, all of which – except for one – come with their own project. The one that doesn't have a project (macroeconomics) overlaps with other courses, because it's a trimester course. Painful.
I'm also trying to convince my professors to be able to do my bachelor thesis on Drupal. By that I mean: develop something that can make Drupal shine in a certain area. (More on that in a later blog post!) That costs its fair share of time as well.
So … I'm extremely busy for school; pulling late-nighters 7/7.

Next, I'm involved in too many projects outside of school: maintaining my Drupal modules (or at least trying to), table tennis, working for Mollom, working in the local AIESEC committee, maintaining DriverPacks.net, and more.

How much is too much? Well, this is too much for me. In reality, I've pretty much only been able to work on school stuff the past 3 weeks. So … to create some breathing room for me again, and because I haven't been able to maintain them anyway, I'm giving the following modules away for adoption:

Carousel, supposed to become the single module for all carousel-goodness (which means it will include the jCarousel and Carousel panel style modules). Me, nor any of my comaintainers have had the time to really push this forward. Porting to Drupal 6 is super easy, patches are already in the issue queue.
Carousel panel style, title says it all. Not much work to maintain.
jCarousel, the jQuery plugin that makes creating carousels easy.
Money CCK Field, title says it all, supports all known currencies and advanced formatting. Not much work to maintain. Needs to be ported to CCK 2 (Drupal 6).
Tabs panel style, title says it all. Not much work to maintain.
Views Saved Searches, needs a couple minor features and needs to be ported to Views 2 (Drupal 6).

This means I'm giving up all my modules for adoption, except for AHAH Helper, CDN Integration and Hierarchical Select.
Let me know via the comments, my contact page or the issue queue if you'd like to become the maintainer of either of those modules.

Finally, I've upgraded my site a bit … which allowed me to provide you with a demo of Hierarchical Select 3! If you plan to use it, or have postponed trying it, you can do so now very easily!

Another awesome DrupalCon!

Wim Leers — Mon, 15 Sep 2008 13:57:20 +0000

It's over. My second DrupalCon. DrupalCon Szeged 2008. I'm posting 2 weeks after date because I had another awesome vacation 2 days after the DrupalCon, to the south of France (in Brandonnet), with friends of the table tennis club.

And it was AWESOME. I met so many new people. I had lots of interesting conversations. Had lots of fun. Saw a different way of life.

AHAH helper module

Wim Leers — Fri, 29 Aug 2008 19:07:50 +0000

AHAH-powered forms were virtually impossible in Drupal 5 (see the note though). In Drupal 6, this is much easier, thanks to the #ahah property. However, it still is really painful to actually use it.

The flaw

You have to write a menu callback for each AHAH-enabled form item of your form. You have to repeat small variations of this piece of code for each callback:

// Build our new form element.
$form_element = _mymodule_add_something_to_form();
 
// Build the new form.
$form_state = array('submitted' => FALSE);
$form_build_id = $_POST['form_build_id'];
// Add the new element to the stored form. Without adding the element
// to the form, Drupal is not aware of this new elements existence and
// will not process it. We retreive the cached form, add the element,
// and resave.
$form = form_get_cache($form_build_id, $form_state);
$form['somewhere']['very']['deep'] = $form_element;
form_set_cache($form_build_id, $form, $form_state);
$form += array(
  '#post' => $_POST,
  '#programmed' => FALSE,
);
 
// Rebuild the form.
$form = form_builder('mymodule_someform', $form, $form_state);
 
// Render the new output.
$subform = $form['somewhere']['choice'];
$output = theme('status_messages') . drupal_render($subform);
 
drupal_json(array('status' => TRUE, 'data' => $output));

Ok, it does make sense. But it takes some time to get used to – too much – and is a treshold that's big enough for many developers to just not implement AHAH forms. It simply shouldn't be this hard.
This current approach of AHAH forms in Drupal is time consuming, hard to maintain and hard to write tests for.

Drupal.behaviors + jQuery: multiple buttons with the same name

Wim Leers — Sat, 23 Aug 2008 14:54:23 +0000

While working for Mollom, I faced the problem of needing multiple buttons with the same name. In my case, this was an absolute necessity on an advanced multi-step form. Sounds super … easy, right? But HTML doesn't support this!

Thankfully, the combination of Drupal.behaviors and jQuery makes it easy to create a work-around! jQuery makes it easy to write the necessary code, Drupal.behaviors makes it trivial to ensure it keeps working even when new content has been added to the page (i.e. after an AHAH callback).

Working at Mollom!

Wim Leers — Fri, 25 Jul 2008 17:29:16 +0000

As some of you may already know, I'm working for Mollom now (summer position).
The reasons for that should be obvious:

It's Drupal work.
I get to work with Dries. No explanation needed here I think.
I get to work with Ben. Don't know him? Remember this: he's really smart and that shows in his machine learning skills and research (he's got a Ph.D. in that field). He made Mollom's algorithms outstanding.

So why I was I hired, without even having an interview?
Well, because certifications don't say much about a person's real skills. They only indicate a base level. Even college/university diplomas are becoming less valuable. It's your real experience that matters. And you can see my real experience in my Drupal.org profile. You can tell that I have a lot Drupal experience, and you can find out the quality of my work by looking in my projects' issue queues and the accompanying code.
That's what really matters.

From day 2, I've be doing daily calls with Dries, and sometimes with Ben too. I really like this way of working, and actually I wouldn't have taken the job if I couldn't do it this way: there's nothing I hate more than losing time because of transportation.
If I have to travel more than one hour per day, I get annoyed. More than 2 hours and I'll just quit.

The first day had both crazy (the morning) and cool times (the rest of the day). So I jotted down what happened that day. To read it, continue after the break.

What you can expect soon…

Wim Leers — Sun, 15 Jun 2008 16:09:37 +0000

I've been very quiet lately, which was especially noticeable on Drupal.org. Right now, I'm in the middle of my exam period (Computer Graphics exam next Wednesday), but as soon as I can (exams will be finished July 2nd), I will resume my Drupal work:

Finish Hierarchical Select 3 for Drupal 5
Port Hierarchical Select 3 to Drupal 6
Finally do that write-up of my svn system for deploying and maintaing Drupal sites (it's ready since last December, just didn't have the time to do the writing).
Finish the consolidation of the jCarousel and Panels Carousel Style modules into the Carousel module.
Make the current CDN Integration for Drupal 5 module scalable (i.e.: stop using a serialized array to keep track of synced files, which was good enough for development purposes, but painfully slow for production).
Also port the CDN integration module to Drupal 6.

Mollom: spam killer minus the annoyance

Wim Leers — Sun, 06 Apr 2008 21:12:37 +0000

I'm late to the Mollom announcing party, because I was on a vacation. Nevertheless, I hope I can still interest some of you with a slightly different angle.

The major issue with spam prevention is that it often (currently virtually always) involves extra steps for normal users. And more steps means less participation. Less participation means less traffic. And less traffic means less popularity, revenue and whatnot. So clearly there is much to be gained to prevent spam without annoying normal users.

Hierarchical Select 2: the developer perspective

Wim Leers — Mon, 03 Mar 2008 12:10:16 +0000

API

The API of the previous version of HS was a beast. Well, not the API, but the implementations. This has been fixed in version 2 of HS: it's now much more elegant and much easier. If you don't have to alter any forms, you can easily implement all hooks in less than a hundred lines, probably even less. The content_taxonomy implementation for example, is about 75 lines if you don't count the form altering. That should make HS much more attractive to other Drupal developers.

Support Hierarchical Select Dynamically

One of the low-hanging fruits is to support HS dynamically (i.e. use hierarchical select form items when HS is installed, use normal selects otherwise).

Hierarchical Select 2

Wim Leers — Mon, 03 Mar 2008 12:09:52 +0000

What is Hierarchical Select?

For those who don't know Hierarchical Select yet, or HS in short, this is a module that provides a new form element. If you're new to Drupal, you may just have frowned upon reading that. A 'form element' in Drupal's Forms API is something like a button, select or textarea element in HTML, or a GUI widget in a GUI.

Now, the goal of HS is actually very narrow: making selections in hierarchies (hence its name) really simple: improve usability. The prime example and candidate for this is of course Drupal's Taxonomy module. The idea is to first select an item from the root level, then pick one of its children (if it has any), then one of the children of the selected child (if it has any), and so on.

The Drupal bank card!

Wim Leers — Tue, 19 Feb 2008 19:36:39 +0000

At the end of 2007, my bank offered free personalized bank cards, meaning that you can put a photo on it. Only hours before the deadline passed (after this deadline, you'd have to pay, bah!), I thought about it. So I quickly started looking for something cool I could put on my bank card.

Drupal 6 released!

Wim Leers — Wed, 13 Feb 2008 16:21:20 +0000

Drupal 6 has been released! So … what's new, what got better and what became easier?

Installation became much easier
There's a new Forms API property: #ahah. This made JS integration with the Forms API possible, even without writing a single line of code! This level of JS integration into the Forms API will result in much more usable Drupal sites, since usability of websites is typically determined by the usability of the forms.
Even more new usability improvements through JS: drag & drop administration for menus, blocks, forums, taxonomy terms and so on.
Multilingual Drupal has been made much easier.
New in core: OpenID!
New in core: actions & triggers, for custom workflows (e.g. send an e-mail when somebody commented on your blog).
New in core: the Update Status module. What? Never heard of it? Your Drupal life wasn't complete, then!
New in core: Schema API, this allows you to write SQL-agnostic database schemas, and you don't even have to write SQL anymore to load/save your modules' objects!
New in core: Batch API, this gives Drupal sites the ability to do processing of huge amounts of information. I think this will result in the creation of many "CMS Foo to Drupal importer" modules.

And that's not even everything … see the announcement at Drupal.org for more details.

Battle plan for Drupal 7: performance

Wim Leers — Mon, 04 Feb 2008 18:11:32 +0000

My battle plan for Drupal 7 is simple: get as many performance improvements or performance-improvement-enablers from my Drupal page loading performance article into Drupal 7. From high to lower priority:

#214934: file_url() and hook_file_server() – yep, patch for Drupal 5 already available and in production use here on this site as well as DrupalBin) – for rule 2, rule 3 and rule 9.
Automatic CSS sprite generator, for rule 1. I still have to play with this to figure out if it's a realistic goal.
GZIP CSS and JS files, for rule 4.
Put JS files at the bottom by default. However, when at least one JS file is added to the header, both jquery.js and drupal.js should be added to the header as well. JS files that alter the appearance of the site heavily (such as carousels) should always be added to the header, and guidelines for this should be written, and should be very clear. This is for rule 6.

CSS sprite generator open sourced!

Wim Leers — Sat, 02 Feb 2008 01:23:11 +0000

I already mentioned the CSS sprite generator in my article on Drupal's page loading performance. More great news though: they open sourced it! See the blog posts by the authors, Stuart Colville and Ed Eliot.

It's under the BSD license though, so it would never be accepted on Drupal's CVS repositories, which is a must. I contacted the authors, asking if they would be willing to dual-license it under the GPL.

The effects of such a module – if technically possible to generate CSS sprites completely autonomously – would be enormous. It would reduce the number of HTTP requests per page considerably: all CSS background images would be aggregated into a single image!
Even on "just" the Garland theme of a default Drupal 6 installation (this includes a logo and a small Drupal banner at the bottom), this would save 7 HTTP requests per page.

Improving Drupal's page loading performance

Wim Leers — Wed, 30 Jan 2008 00:33:14 +0000

Introduction

Google dominates the search engine market for a large part thanks to its spartan, no-bells-nor-whistles interfaces. But also thanks to its incredible speed (which is partially thanks to that spartan interface, of course).

Since you're reading this article, you're probably a Drupal developer. It's pretty likely that you've had some visitors of your Drupal-powered web site complain about slow page load times. It doesn't matter whether your server(s) are shared, VPSes or even dedicated servers. Visitors that live abroad – i.e. far from where your servers are located – will face the same performance issues, but at even worse scales.
This article is about tackling these issues.

TextMate command to look up functions at api.drupal.org

Wim Leers — Mon, 26 Nov 2007 07:01:48 +0000

Is it also for you a routine to look up the documentation for Drupal hooks at api.drupal.org?
If you also use TextMate and are sick of having to command-tab to your browser to get to the documentation, then you'll have a much better alternative in about 15 seconds.

Go to Bundles > Bundle Editor > Show Bundle Editor. There, click the plus-button in the bottom left corner to add a new command. Set the input to None, the output to Show as Tool Tip and the scope selector to source.php. Copy/paste the command below and assign a shortcut -- I use CTRL + D.

if grep <<<${TM_CURRENT_WORD:-!} -Esq '^[a-zA-Z0-9_]+$'
	then
		FILENAME=${TM_FILEPATH##*/}
		BASENAME=${FILENAME%%.*};
		if [ `echo "$TM_CURRENT_WORD" | grep -E "${BASENAME}_[0-9A-Za-z_]"` ]
		then
			HOOK=${TM_CURRENT_WORD##${BASENAME}}
			URL="http://api.drupal.org/apis/5/hook${HOOK}"
		else
			URL="http://api.drupal.org/apis/5/${TM_CURRENT_WORD}"
		fi
		exit_show_html "<meta http-equiv='Refresh' content='0;URL=$URL'>"
	else echo "Nothing to lookup (hint: place the caret on a function name)"
fi

If you now place the cursor above a Drupal core function and press the shortcut, you'll be presented with a direct lookup of that function at api.drupal.org. Better even, if you place the cursor on a hook implementation, e.g. mymodule_menu, then you'll get the documentation of hook_menu. Note that your file must be saved for this to work (and it must be named according to Drupal conventions, i.e. mymodule.module).

CVS diff with new files: fakeadd

Wim Leers — Tue, 30 Oct 2007 09:53:16 +0000

Ever been disgruntled by the fact that you can't do a cvs diff with new files, because you haven't got write access on a Drupal contrib module's CVS repository and thus can't cvs add those new files? There's a solution though: edit CVS's Entries file. But who likes manually modifying files over and over again?

The solution: fakeadd! (I've attached the file to this post in case this site ever goes down.) This nice shellscript allows you to update the Entries file using a simple syntax: fakeadd newfile.php.

A quick install how-to:

Save the file to a location of your choice, I've saved it in ~/scripts.
Open your ~/.bash_profile file (if you're using the bash shell, this is the default shell on OS X) and add the following:

# Add custom scripts, such as the CVS fakeadd script. export PATH=$PATH:~/scripts
Finally, make the script executable: chmod +x ~/scripts/fakeadd.

New module: Hierarchical Select

Wim Leers — Sat, 27 Oct 2007 14:05:42 +0000

Almost 3 weeks two months ago now, I have released the Hierarchical Select module. (This post was originally written during the DrupalCon, but time has been sparse since then and bugs were there, which is why I'm posting this so late.) It defines a new form element: hierarchical_select. It makes the selection of something in a hierarchy much more usable. By default, selects of the taxonomy module (on node forms) are forced to use hierarchical selects - if the vocabulary is hierarchical. The same is applied to content_taxonomy CCK fields, when the select widget is used.
Note that the hierarchical select form element does not support the selection of multiple items. And obviously, it only selects the "deepest" item.
I'll immediately skip to the demo, because a live demo says so much more than words.

And it has an API!

This part is only interesting to people who want to make use of this form element for other modules than taxonomy.

The module comes with an API, so that any module that manages some sort of hierarchy, can define a rendering function that will be used to render the hierarchical selects. It can't be done completely automagically, because it varies greatly from case to case what input data is necessary to generate the necessary tree structure.
For example: taxonomy needs a vocabulary id and the id of the currently selected term (term id). Another module may only require one id, while another may require 3 ids. You will always get the id of the newly selected item in any of the selects back (which is the term id in the case of taxonomy).

hook_hierarchical_select_render($hsid, $selection, $params)

$hsid - hierarchical select id, will just be used as a unique identifier in the HTML
$selection - the value of the item that was selected by the user
$params - an array of parameters that are passed around, these are considered "static", e.g. the vocabulary id

You can look at modules/taxonomy.inc for an example implementation. See API.txt for details.

Future: Drupal 6 and CCK 2

There's room for improvements in the API. But that'll be for a second version, which will be Drupal 6 only, because of the interesting change in CCK 2: CCK widgets will simply be FAPI form elements. That means that we'll be able to use the hierarchical select form element for several CCK fields!

Post-Pre-DrupalCon Social

Wim Leers — Wed, 19 Sep 2007 07:50:02 +0000

Last night we had the Pre-DrupalCon Social near La Rambla. For me, it were a lot of new faces (it's my first DrupalCon after all). Among them were the most infamous Drupal developers: eaton, webchick, jeff (well, all Lullabots really), chx, Heine and so on. Geeky stuff, good times!
After some cosy chatting, we got to dinner. But apparently the venue, Muebles Ciudad, only offer Japanese food. So it was a shot in the dark for me, and the first time ever I had to eat with sticks. I finished much later than the rest, but at least I managed to eat something :P. I don't think I'll eat it again any time soon though. French fries ... hmmmmm.

At about 10 PM we returned back to the hotel, because Larry "Crell" Garfield, my roommate, still had to finish the code examples for his Drupal and PHP 5 presentation of today. Since chx will also be talking in that presentation, he joined us, so they could go through the presentation -- and to show off his brand new Simpletest Automator module of course :). It's a really powerful tool that will hopefully make unit testing in Drupal a common thing.

New in Panels 2: Mini Panels and Panel Styles

Wim Leers — Fri, 24 Aug 2007 00:15:36 +0000

A couple of weeks ago, I agreed to work on the Panels Mini module, which would become part of the second version of Earl Miles' outstanding Panels module. While working on that, I suggested to let modules implement styles for panels. (I was inspired by Nedjo Rogers' Panelsblock module, which sort of gives you "tabbed Mini Panels" for Panels 1). All you then would have to do, is: pick the Panel Style in the settings for the Mini Panel (or whatever other Panels implementation chooses to support it) and then change the CSS to get exactly what you want. This should prove to be a real timesaver.

More about Panel Styles

I've written 2 Panel Styles so far: Carousel Panel Style and Tabs Panel Style. The latter might be included in Panels 2. Both were sponsored by Paul Ektov.
To write a Panel Style, you only have to implement 2 hooks, or 4 hooks if you want to offer configuration options. Hopefully that will result in many contributed Panel Styles!

hook_panels_panel_style_info(): returns an array of Panel Styles that the module implements, along with some metadata
hook_panels_panel_style_render(): receives the Panel Style configuration and an array of (rendered) panes, returns the rendered HTML
hook_panels_panel_style_settings_form(): (optional) defines a form if your Panel Style supports some configuration options
hook_panels_panel_style_settings_validate(): (optional) validation function for the form of the previous hook

(Fancy?) Demo

Without further ado, a fairly fancy demo of all this.

I want this yummy stuff too! Where can I get it?

Mini Panels and the Panel Styles hooks are available in the Panels 2 CVS, and will be included in Panels 2 alpha 8 when it gets released. Be warned that Panels 2 is still a work in progress!