mr-drupal: simpler Drupal code updating & deployment
What?
This article proposes a novel, simpler way of managing Drupal sites (where “managing” in this case is solely code updating & deployment).
It relies on only one command-line tool: mr
(“a Multiple Repository management tool”, http://joeyh.name/code/mr/), plus a Drupal plug-in for that tool: mr-drupal
, https://github.com/wimleers/mr-drupal.
Why?
If you run Drupal sites, you need some way to manage them; some way to keep them up-to-date. As of Drupal 7, there’s a built-in update manager, but it doesn’t use a VCS.
Most likely, you want use a VCS to manage your Drupal site. You may be downloading tarballs and checking their contents into your VCS, or maybe you’re using git submodules or even git-subtree
. For all of these, there’s a whole lot of process, a lot of steps, a lot to learn, and a lot of tricky things you have to think about each time you want to update something. Too much that can go wrong.
That is fine for big, commercial, team-backed sites. But it’s a pain to do for smaller sites, that maybe only are updated once every few months. When you update rarely, it becomes very easy to forget about one of those tricky things. And then, for fear of doing X or Y wrong, which might break your site, it causes you to update your site even less often.
I probably evaluated everything out there1. All of it was too complex for my needs. I want my code deployment system to be simple and preferably Drupal-agnostic.
It’s that niche that this approach to Drupal site management attempts to simplify.
How?
mr
is a small, stable shell script (written in Perl, unfortunately) to help manage multiple (VCS) repositories simultaneously. By using the Drupal plug-in provided by this project, you can leverage a subset of mr
’s functionality to simplify Drupal site management.
The Drupal plugin offers you a simple, declarative syntax to automatically clone and update git repositories for each Drupal module you use from git.drupal.org
.2
There’s almost nothing to learn. Hence, there’s almost nothing to forget. There is a single configuration file that is (mostly) declarative, so if you look at it again in 2, 6 or 12 months, it will still make sense.
Essentially, you only need to know two things:
- the desired state is declared in the
.mrconfig
file - get to the desired state by running
mr update
Note that the tool/approach described does not cover running database updates etc. (though it is possible to integrate that as well). I prefer to do this manually on smallish sites.
Getting started
Steps to be repeated on each machine:
- Install
mr
. (Package in Debian, unofficial RPM and on OS X viabrew
.) - Install the Drupal plug-in for
mr
: copy thedrupal
file in themr-drupal
repository into/usr/share/mr/
.
Creating a config file for your site
Steps to be repeated for each Drupal site:
Create a
.mrconfig
file (don’t forget the leading period!), it should look like this:# Use the Drupal plug-in for `mr`. [DEFAULT] include = cat /usr/share/mr/drupal # Drupal itself. [www] project = drupal version = 7.19 # Set the proper file system permissions on all Drupal files. The owner is # set to the current user, the group is set to www-data. # See http://drupal.org/node/244924. fixups = drupal_set_permissions `whoami` 'www-data' # A branch of a Drupal module. [www/sites/all/modules/cdn] project = cdn version = 7.x-2.x # A tag of a Drupal module. [www/sites/all/modules/comment_notify] project = comment_notify version = 7.x-1.1 # Custom themes live in a non-git.drupal.org repository. `mr` is smart enough # to automatically know this is a git repository, and knows how to update it. [www/sites/wimleers.com/themes] checkout = git clone git://github.com/wimleers/wimleers.com-themes.git themes
As you can see, it’s easy to mix in custom modules/repositories.
- Run
mr update
. Your Drupal site will be built based on your.mrconfig
file, relative to where the.mrconfig
file lives.3 - Optionally check in the
.mrconfig
file into a VCS, so you can roll back to previous versions. Highly preferable, but not essential.
In the above example, we leveraged one “advanced” feature of mr
: the ability to do “fixups” (after each checkout/update). In this case, to guarantee correct permissions using the drupal_set_permissions()
function. mr-drupal
also includes a git_apply_patch()
function to git apply
patches (it also automatically detects whether the patch has already been applied).
Note that there is nothing Drupal-specific about any of this! You can check out Drupal modules at www/sites/all/modules/
, a static “status” site at www/status
, HTML5 presentations at www/talks/
, and so on. Things won’t break because Drupal or Drush are updated.
To update your site
Modify the
.mrconfig
file: change version numbers, add modules. Remove a module by adding a line like this:deleted = true
.Run
mr update
. That’s it!4
Advanced: environment-specific (Dev vs. Prod) actions
If the above doesn’t look like it would cut it for you because you need to set up things differently for your local development setup and your production server, then you’re right (i.e. DTAP).
To easily deal with that too, this drupal
extension for mr
provides three functions:
whoami()
— a simple wrapper around thewhoami
command.hostname()
— a simple wrapper around thehostname
command.on()
— uses the two functions above to be able to write something likeon 'wim@wimleers.local'
, which evaluates to true if those are indeed the current user and host.
You can then upgrade the above example of
[DEFAULT]
include = cat /usr/share/mr/drupal
[www]
project = drupal
version = 7.19
fixups = drupal_set_permissions `whoami` 'www-data'
to something like this:
[DEFAULT]
include = cat /usr/share/mr/drupal
lib =
get_dtap() {
if on 'wim@wimleers.local'; then
echo 'development'
else
echo 'production'
fi
}
drupal_fs_group() {
if [ "$(get_dtap)" = 'development' ]; then
echo 'staff' # Mac OS X
else
echo 'www-data' # Linux
fi
}
[www]
project = drupal
version = 7.19
fixups = drupal_set_permissions `whoami` `drupal_fs_group`
In lib
, you can define functions you want to use. In this case, we defined the get_dtap()
and drupal_fs_group()
functions. The latter uses the former.
Now the group used for enforcing the correct permissions is set based on the environment.
But really, you can do whatever you want: anything that can be used in a shell script can also be used here. This is the part of mr-drupal
where you can make things as complex as you’d like. The complex parts are opt-in.
Conclusion
One file (.mrconfig
) and one command (mr update
) together cover 90% of the smallish Drupal site management needs.
Easy to understand, very few dependencies, extremely customizable.
Simple for simple deployments, complex for complex deployments. Instead of always complex.
P.S.: I’ve been using mr-drupal
to keep my sites updated for the past several months without problems. Rather than spending >1 hour (to update locally, then look up drush
commands and triple-check everything before calling drush rsync
), it now takes me just a few minutes to update a site!
-
I evaluated the following:
- Drupal core’s recommended git practices which involve checking in contents of tarballs: implies loss of version history and duplicates code
dog
— short for “Drupal on git”: unfortunately too immature, because this got me very excitedgit
submodules: too many steps to add or update modules, too easy to forget to pull in submodule updates, nightmare to deal with patchesgit-subtree
: even more complex thangit
submodules- Drush Deploy: only handles the effective “deploy” step, still requires either of the above
git
strategies drush make
: took years for it to “kind of” support updates aka “remakes” (which inspires little confidence), it is highly Drupal-specific and personally I have very bad experiences withdrush make
-
As is the case for any site that has its modules on disk as
git
repositories: you must use the Git deploy module to ensure Drupal core’s “Available Updates” report continues to work. ↩︎ -
Speed it up by telling it to work in parallel:
mr -j 10 update
would do up to 10 updates in parallel. ↩︎ -
The only exception: those that are marked as deleted.
mr
does not actually delete them for you. For example, if you’ve marked thedevel
anddrupad
modules as deleted and thedevel
module no longer exists, it’ll output something like this:$ mr --stats update mr update: /htdocs/wimleers.com/www/sites/all/modules/cdn Updating 'cdn' to version 7.x-2.5 (from 7.x-2.3)... Done. mr error: /htdocs/wimleers.com/www/sites/all/modules/drupad/ should be deleted yet still exists […] mr update: finished (29 ok; 1 failed; 1 skipped) mr update: (skipped: /htdocs/wimleers.com/www/sites/all/modules/devel/) mr update: (failed: /htdocs/wimleers.com/www/sites/all/modules/drupad/)
The
devel
module is skipped (no error!) because it’s marked as deleted and doesn’t exist. Thedrupad
module is marked with an error message (because it’s also marked as deleted but still exists). ↩︎
Comments
That’s very, very interesting!
You’ve done the job I’ve been postponing doing myself for way too long…
+1000 for listing what you did evaluate and why it didn’t work.
This I really would like to look into more, looks really intresting. Thinks for telling.
You’re welcome :) Looking forward to concrete feedback now, and pull requests!
Thanks for this Wim. I’m doing a similar exercise at the moment.
Why did you decide that git subtree was more complex that submodules? To me it looks less complex.
I presume you discounted Aegir because you have to have dedicated server/s?
One thing I really like about your approach is the declarative nature. I always want meta information that is separate from the information under control. It makes it easy to know if there have been changes which have not been processed; and, it lets you keep a history of changes. That’s one of the I like about using git. What I don’t like about git all the other systems I’ve found is the lack of space for information about the ‘special stuff’. The infrequent things you do for a particular site which need some sort of special attention - e.g. not updating a particular module.
The reason I consider
git-subtree
to be even more complex is partially because it’s something that’s not part ofgit
itself, and is a similar series of extra commands you have to use.I discounted Aegir because it’s designed for another level/layer: it’s intended to manage a network of Drupal sites and even sets up web server configuration etc.
Can you clarify the “special stuff” part that you’re talking about? That’s not entirely clear to me.
Since 1.7.11 git-subtree has been part of mainline git, I think. That’s May 2012. Yes, it’s an extra set of commands in that you have to notify git when you are adding a Drupal module - you add a remote repo for the module pointing at drupal.org, then you do something like this to add a module:
git subtree add -P docroot/sites/all/modules/devel upstream-drupal-devel --squash -m "Adding in Devel 7.x-1.3" tags/7.x-1.3
After that if you make patches to devel, you have to remember when to use -P docroot/sites/all/modules/devel to notify git that the change applies to the devel module. So you have to have a commit for changes to the module. But you probably would anyway because a change to a module is a discreet element.Where it falls down:
Special stuff: It’s the mixture of stuff you forget is special about a site when you don’t deal with it regularly. E.g. Remembering there’s a patch to a module that’s not worth getting into the drupal.org version of the module because it’s site-specific; remembering Hacked is installed to remind you about a change like that; holding back on upgrading a particular module because later versions lack some features - e.g. the D6 nodewords module that lost token support; site-specific rules to remember - e.g. a site I worked on a few years ago had its own template engine for historical reasons - before a module could be updated it needed a diff to check whether the module template files had changed - and if they had, then it needed work in preparation for the update.
When you use drush you don’t have a place for info like this. Neither does normal git. But mr does - the .mrconfig which you have to edit to update.
Caveat: I’m still experimenting with git subtree workflow, and I’m pretty sure I haven’t discovered all the gotchas.
Right, even when
git-subtree
is integrated in mainlinegit
, it is still too cumbersome to use.git-subtree
user to manually modify eachgit-subtree
‘d-in Drupal module to add the module’s version number to the module’s.info
file. That’s of course a rather insane requirement.RE: “specialty stuff”: exactly! For this very site (<wimleers.com>), I have the Google Webmaster HTML file and one or two Drupal core patches in the git repository where I keep the
.mrconfig
file for this site. I then just letmr
respectively copy it into the appropriate location and apply the patches :)Thanks for your feedback — your point about “no good place for the metadata/special stuff” is a good one that I failed to mention!
Wow Wim! Thanks for sharing mr-drupal and giving such a great intro to mr. This looks very promising.
Thanks :)
And thanks for the mention in The Weekly Drop — it brought in a lot of additional attention and a bunch of useful comments :)
Hey wim: if this strategy depends on git_deploy, then anyone testing it should be paying attention to this issue [as I see you are already]:
http://drupal.org/node/1511112
The short version is - git_deploy has a performance problem [e.g. it’ll take about a minute to show your modules page once you have any significant number of contrib modules in there].
Yep, it has a big performance problem, but it’s also solvable for sure :)
Very interesting! A couple questions:
I can answer that one from experience. If you have customizations of even one module and one theme, a regular (let’s say monthly) automatic Drush update can mean monthly havoc for your site. If done every 6 months, it means in 6 months you have to be poised to undo and repair whatever the auto -update caused. Having it pre-scripted can be very handy so it will just avoid the custom modules. And if nothing else, the script itself serves as a reminder to revisit the customizations to see if they’re still necessary.
It was high on my to-do list with my admin to find an alternative to automatic Drush updates, so, Wim, thanks!! I’ll be bringing this to his attention!
That’s what drush make is for. Mr. Is effectively equal to drush make
@Mike: I agree with Diane, running
drush up
automatically is very dangerous.(This answers question 2 in your comment.)
Dalin already answered question 2. I’ll answer question 1 here.
I discount
drush make
for several reasons:Combine all of the above, and it simply boils down to this: I don’t trust Drush for deployment. Sure, I could start to experiment with it again, possibly Drush 6 is better and hence sufficiently reliable for doing deployment. But Drush only becomes more complex, so it becomes increasingly hard to evaluate it.
Drush can be used for updating. See https://drupal.org/sandbox/grndlvl/1930260 We’ve been using this in-house for all of our major projects and it’s great. We have a build server running on every commit to Git that uses make-sync to confirm that what is committed in Git matches what is defined in the make file.
I haven’t really experienced much breakage with Drush moving up in versions. But I think that’s because we’ve been fairly conservative - we have a ‘standardized version’ that is on all of our machines and we don’t upgrade it very often.
Drush deals with non-Drupal stuff with ease (if by “stuff” you mean 3rd-party libraries.)
I am 100% with you on the complexity issue though. I’ve done a bit of work on Drush plugins and I always get the feeling that something is really wrong, mostly with the argument system. But it’s big because it does a lot more than just library/project management. Mr. can’t transfer DBs from server to server with some sanitization thrown in (well it could, but it would require an equal amount of code).
You mention the word “deployment”, but that’s not really what we’re talking about here. That’s a completely different problem, but I agree — I don’t see Drush as being useful there.
I don’t think I ever said
drush make
can’t be used, did I? :)make_sync.drush.inc
is hardly trivial at >650 LoC that are extremely hard to understand without deep Drush knowledge.I see a lot of value in staying within the Drupal/Drush ecosystem. I’m not opposed to Drush. I’m only saying that Drush/
drush make
don’t work for smaller use cases — I said this in the article:And in fact, many of the things are in exact line with my argument:
All of those points require a big team, because a one-person project is never going to figure out/build all of the above!
Actually, I mean anything. Like e.g. cron scripts, HTML5 presentations, a subsite. Anything. Much more than 3rd party libraries that are used by Drupal modules.
Finally: yes,
mr
andmr-drupal
don’t solve the whole deployment problem, they only aid in solving the code deployment problem. In my case, it’s good enough to deal with DB dumps and moving those around manually. Small sites have no need for two-way content syncing and all that.It’s not just that Drush provides both code and DB deployment, it’s that it does even more than that. That’s why it’s so complex. That’s why it’s so big. That’s why it’s scary to work with.
If something pops up that’s equally simple as
mr-drupal
, or only slightly more complex, and it deals with DB deployment, then I’d be happy to switch to that. I don’t see that happening soon though.Even when that does happen, I’ll still be able to use
mr
for handling other things — it is in no way specific to Drupal or even web development. Many people use it to manage their home directory, for example.I added a small pull request, it really need some work, and I doubt the approach is the best on, but better to have something to talk about :-)
So you must be referring to https://github.com/wimleers/mr-drupal/pull/1 :) Thanks for starting that! But, indeed, I think that’s not really the right approach. For starters, you’ll run into problems when updating a module from one version to the next, because you’d have to clean up the module’s
.info
file. I think it’s better to just fix the performance problem in thegit_deploy
module: https://drupal.org/node/1511112.Have you tried Kraftwagen? It uses a combination of Drush Make with the ability to add or exclude modules based on environment (dev, stage, prod etc.) with automated configuration.
It does require a specific and unusual project structure though, so probably not a simple solution for small sites.
I can’t say that I have. I saw it pass by, but I completely forgot about it. Thanks for reminding me!
Kraftwagen again seems too complex for simple sites. Too many steps to remember. It also aims to do content & config migration/syncing, which is much, much more than
mr-drupal
does.Also, config syncing will be a solved problem in Drupal 8, so a big part of what Kraftwagen does will become irrelevant then.
I’d say that
mr-drupal
should not be used for sites that need content migration. For those, Kraftwagen seems a much better solution. It is possible to usemr-drupal
in these situations too, but then you’ll need to think of a content migration strategy yourself — or maybe just use that part of Kraftwagen (though I’m not sure if that’s feasible).While there is no real silver bullet when it comes to deployment, I think this is a very handy tool for small to medium projects.
The big advantage here is that there’s less code to worry about as the state of a site instance gets dictated by a declarative config file, not by the code checked into the project repo. It does remind me of tools like Chef or Puppet, which pretty much implement the same concepts. :-)
Quick questions:
- .gitignore
- .mrconfig
- .htaccess (if modified)
- sites/*/{modules|themes}/{custom|contrib_patched}
Am I missing anything else?I’ve yet to implement it in a workflow, but this looks quite promising.
For me, the project repository contains:
.mrconfig
(duh).gitignore
file, because it doesn’t seem very useful to me at the.mrconfig
level?.htaccess
file, because that should be a patch — Drupal core sometimes modifies its.htaccess
file, so you should patch Drupal’s, so you get the latest changes (or maybe the patch will fail if it no longer applies)mr-drupal
includes thegit_apply_patch
function, which you can use like this:And, indeed, the
.mrconfig
file can be used to automate many things. Patches are already covered (see above), but permissions are, too! For that,mr-drupal
includes thedrupal_set_permissions
function, which you can use like this:Another thing that is covered: removing the unnecessary
.txt
files likeCHANGELOG.txt
,README.txt
, and so on:mr-drupal
provides thedrupal_rm_unnecessary_txt
function for that.So, maybe a more complete example of the more advanced things you want to do is this one (copied from the
.mrconfig
file for http://driverpacks.net):So: file permissions, two core patches, a Google site verification file and a symlink at the root of the site to have nice links for “userbars”. Only the first is highly Drupal-specific. And only the first calls yet more functions — to see how to do that, read the “Advanced: environment-specific actions” section of the article again :)
I love the concept. It is much more development-centric (and space friendly!) than an endless constant (and slow) Ægir/platform/make loop, and I feel that the agnostic approach makes for a more straightforward integration with automated build tools.
I am implementing this model, but considering a few changes for a fully clustered environment with multiple contributors and a git flow-type workflow and would value your input:
mr
build will be committed into a “docroot” directory in themr
repository, so the same repository can be used to keep a cluster synchronized. Files will be stored/linked elsewhere.git subtree
.git subtree --squash
will be used on third-party repos to minimize noise.mr
would be setup to pull from the local projects/ subdir repositories.My thoughts are that while this does make the repository larger and more expressive, I am hoping it provides:
git
) for quick reversal and precise reproduction (local debugging)I suppose you might archive and purge at some regularly interval for very large projects?
I am curious if you have any recent updates or anecdotes on your experiences with
mr
andmr-drupal
or insight on some of the concepts above.Regards, Jason
I’m glad you like it :)
What you describe sounds sensible and doable. The mentions of “git flow”, Ægir,
git subtree
and so on imply to me that you’re working on a large-scale platform.I personally have no need for those complex setups, but you can definitely can combine
mr
with them. I’d say: just be careful to not make things overly complex.I only wonder what you mean by & purge what?
. ArchiveAs for recent updates or anecdotes: I still use
mr
+mr-drupal
to maintain three websites, and it still is significantly easier than the frustrating process I had to endure whenever I had to use Drush. One is on Drupal 6, two are on Drupal 7.Finally: please ping me when you have news to share, I’d love to read how you’re using
mr-drupal
:)Will do. Thanks again.
Using subtrees without
--squash
, commit history from subtrees is included in the main project. If there are many subtrees and you are interested in aggregating the commits in this way, then you might want to manually squash/collapse the history at certain points (yearly) to trim the repository.A reasonable walkthrough of a process like that is here: http://stackoverflow.com/questions/250238/collapsing-a-git-repositorys-h…
I’m confused. Or you are. Or we both are :)
Part of the point of
mr
is that you have a separate git repository for each project. A single Drupal module would correspond to a single git repository.mr
can be used to check out (orclone
ingit
parlance) put each such repository in the desired location, without having to import each such repository’s history into a “main” repository.If you’re going to trim the history yearly, then you will lose valuable historical context when debugging (and particularly when
git bisect
ing).So either I’m confused by how/where you want to use
git subtree
, or you’re confused about howmr
sidesteps the need forgit subtree
in the first place. Or maybe I’m just plain confused :)Right. The difference is that I intend to git subtree the custom modules into the mr repository to consolidate the ‘development environment’ while still maintaining the custom modules in their own repositories. As a developer, you will have the choice of checking out just the module or the entire development snapshot. If you only check out the mr repo, you will have included with it the remote repos as subtrees and can modify/commit/merge/etc as normal. This has the affect of aggregating all related commits into the history of the mr repo, which is desirable from the standpoint of ongoing code reviews/audits for an entire project.
So in this scenario:
Now the merged development build is checked into the mr repo, and any cluster server that wants to serve the Develop branch can pull without a mr rebuild.
Project/module gets updated in either the ‘stand-alone’ repo or via the mr repo’s subtree, committed, etc. Run mr rebuild and add/commit the new build snapshot.
mr builds should go much faster, because the repo/dependencies are already local via the subtree.
Tested/staged builds would then be pushed to the Master branch for deployment to production.
I disagree. The
mr
repo should only refer to tags of actual code repos. Themr
repo should only be the glue.If you’re going to do “actual” development for each Drupal module in its own repository, yet still include the most recent history of each Drupal module using
git subtree
, then you’re just signing up for a lot of overhead work.I can see why one might want to be able to have a single repository for all sub projects (i.e. Drupal modules and so on), but I personally don’t need or want it. However, if you want a single repository to rule them all, then the value of using
mr
/mr-drupal
has mostly evaporated. The simplicity I tried to achieve has vanished. Because now it’ll be up to you to manually sync repositories!For example, imagine you’re working on a
foo
Drupal module. You’re working on it within themr
repository’sgit subtree
‘d in version of it… so now you have to manually sync those commits to thefoo
Drupal module’s own repository. Or vice versa.Far too large a margin for error, IMO.
It’s much, much simpler to just tag a new version of the
foo
Drupal module and then update your.mrconfig
file to use that instead. Or, if need be, include a patch forfoo
in your.mrconfig
(e.g. if it’s not a Drupal module you own).Tagging in each subproject’s repository and then updating the
mr
repository to use those tags. That’s much simpler. That keeps each subproject’s history sane. That ensures themr
repository’s history is both simple and … a nice high-level change log of what has changed in the main project.Of course, that is just my opinion, and maybe I’m misunderstanding you, but it sounds like you’re making things more instead of less complex :)
I use a little drush script to update 10 sites. Probably will hook it up to cron as it worked fine so far.
# Download latest stable release using the code below or browse to github.com/drush-ops/drush/releases. wget http://files.drush.org/drush.phar # Or use our upcoming release: wget http://files.drush.org/drush-unstable.phar # Test your install. php drush.phar core-status # Rename to `drush` instead of `php drush.phar`. Destination can be anywhere on $PATH. chmod +x drush.phar sudo mv drush.phar /usr/local/bin/drush # Enrich the bash startup file with completion and aliases. drush init /usr/local/bin/composer self-update #cd /usr/local/src/drush/ #git checkout -- . #git pull && composer update echo NOW UPPING SITE 1 cd /home/..../public_html su -c "cp .htaccess backup.htaccess" myuser su -c "drush -y rf" myuser su -c "drush -y up" myuser su -c "cp backup.htaccess .htaccess" myuser su -c "drush -y updb" myuser su -c "drush en upgrade_status -y" myuser
and the rest of the sites come here