Automating software development and deployment

Phing is a tool designed for automated project building based on Apache Ant distributed as a PEAR extension. It provides the user with the possibility of building the project /...

Phing is a tool designed for automated project building based on Apache Ant distributed as a PEAR extension. It provides the user with the possibility of building the project / application in the same manner as make does with its Makefiles, making complex process execution (which is commonly human error prone) much easier.

Phing’s authors highlight its main features on the project’s page:

  • Simple XML buildfiles
  • Rich set of provided tasks
  • Easily extendable via PHP classes
  • Platform-independent: works on UNIX, Windows, Mac OSX
  • No required external dependencies
  • Built for PHP5

In addition they’ve provided a rich and clear documentation both for the building mechanism and the API of PHP classes whom which the extension is constructed.

Getting started

The cost of integrating Phing into the project is so small, that it can be done at any stage of the project – regardless we’re developing our own solution, or use existing codebase.

After defining project requirements and extracting frequently repeated or requiring high precision operations one can transform them into a buildfile using defined XML syntax and let the rest for Phing to handle.

In order to start working with Phing (after installing the PEAR extension) all you need to do place among the project files a single XML file called build.xml, which will contain the builder’s targets (different file name can be used using the -f switch). Next while in the same location as the buildfile you simply execute the phing command.

An example project layout containing the buildfile has been shown on the figure below.

Example of the buildfile’s placement

At minimum the file should contain a project tag with at least one build target which can be represented as a sequence of tasks (task) which are Phing’s basic executable element. Different targets can depend on each other – the depends attribute lets you define those dependencies – so we’re ensured that before building a particular target all required side targets will be built before.

Process modeling lies in identifying particular targets and describing the way of its accomplishment using tasks executed in a defined sequence. Phing also provides the possibility of applying conditional constructs letting you change the build process depending on the environment (for example depending on the host OS different libraries or software can be used) what makes this tool yet more versatile.

Example applications

The buildfile

We assume completion of targets listed below:

Target Description
permissions Setting up proper file permissions
configuration Copying the configuration file’s template and placing proper values defined in the build.properties file into the copy
setup-db Execution of SQL code from a file designed for particular RDBM using PDO with the host and login data stored in the property file

The code below presents an example build.xml file which preforms tasks described above.

<?xml version="1.0" encoding="utf-8" ?>
<project name="example" default="development">
 
	<!--
		Loading the property file (property = value)
		allows configuring the build process without
		changing the buildfile XML code
		 
		The value stored in the property file can be
		obtained using ${property}
	-->
	<property file="build.properties" />
	 
	<!-- Sets up directory permissions -->
	<target name="permissions" >
		<chmod file="./assets/" mode="0775" />
		<chmod file="./protected/runtime/" mode="0775" />
	</target>
	 
	<!-- Prepares the application's configuration -->
	<target name="configuration">
	 
		<!-- Copy and overwrite the file -->
		<copy file="./protected/config/main.template.php"
			  tofile="./protected/config/main.php"
		      overwrite="true">
			<filterchain> <!-- Copy using a filter... -->
				<expandproperties /> <!-- ... expanding properties -->
			</filterchain>
		</copy> 
	</target>
	 
	<!-- Sets up the databse using SQL files -->
	<target name="setup-db">
		<pdosqlexec
			url="${app.database.connector}:host=${app.database.host};
				 dbname=${app.database.name}"
			userid="${app.database.user}"
			password="${app.database.password}">
			<transaction src="data/schema-${app.database.connector}.sql" />
		</pdosqlexec>
	</target>
	 
	<target name="development"
			description="Builds the development version"
			depends="permissions, configuration, setup-db">
		<!-- ... -->
	</target>
 
</project>

The property file

For the sake of simplicity in working with Phing or simply configuring the build process we can use the property files containing property = value pairs, which can be referenced from within the buildfile. Some example assignments used to configure a database connection using PDO are shown below – the file could also contain deployment FTP server connection data and login creditentials.

# Database type
app.database.connector = pgsql

# Database server
app.database.host = localhost

# Database server port
# MySQL = 3306; PgSQL = 5432
app.database.port = 5432

# Database name
app.database.name = db_name

# Database user
app.database.user = my_webapp

# Database user password
app.database.password = ~t0p!S3CreT~

It’s worth remembering that the buildifle can include information from any number of property files and also that using a buildfile is optional. All mentioned values can be stored directly in the buildfile using the property task.

Execution of the build process

In order to preform the build process – as mentioned earlier – all we require is access to the shell and issue the phing command optionally adding the target name as the parameter. If no target is explicitly specified Phing will attempt to build the default target defined in the default property of the project tag.

# Builds 'permissions', 'configuration',
# 'setup-db' and 'development'
$ phing

# Same as above
$ phing development

# Only selected target - setting up permissions
$ phing configuration

# Only the DB installation
$ phing setup-db

Filters and execution controll

In the presented buildfile we’ve stored a copy task using the filter chain containing the expandproperties filter. The task of this filter is to substitute referenced properties visible within given scope in the file being processed. The property file was included at the project level (not a single target) so the properties contained in that file are visible from within every project’s target.

The mechanism in relation with the configuration file’s template shown below results with creation of a ready-to-go database connection configuration file. In case of systems based on available – third party – components it’s commonly required to configure each of them separately. Thus creating a series of configuration file templates (each for a particular component) and embedding the configuration process in the build file gives the possibility to rapidly configure / reconfigure the application as a whole, or components one by one at any time. To achieve this, one has to update the property file contents and run a proper phing target.

<?php
    return array(
		'connectionString' => '${app.database.connector}:host=${app.database.host};'
							  . 'dbname=${app.database.name}',
		'user'             => '${app.database.user}',
		'password'         => '${app.database.password}',
	);

Phing has been armed with a variety of filters allowing (for example) converting tabs to spaces, PHP comments removal or substituting strings in files based on regular expressions.

More importantly Phing contains many tasks, allowing the programmer to model any build process, including user console input or Git / SVN integration.

Extensibility

Phing is a tool developed using PHP. Despite the variety of functions provided by its authors it doesn’t cover all possible application scenarios for tools of this kind. That’s why the authors have left a “backdoor” for more demanding users – a object oriented thus extendable architecture.

If Phing doesn’t have a particular functionality out of the box, there’s nothing stopping a programmer to add a new task which is a typical PHP class extending from the Phing’s Task class – refer to the official docs.

Why should I use it?

After many struggles comes the time when the project moves from development to testing, and further to production environment. In case of long term and expandable projects, or those filled with bugs (so always) comes the time for an update.

On each of those steps the accuracy of all the taken actions is crucial. One can not simply take the company’s intranet offline for a couple of hours since it may cause serious malfunctions or even paralyze the whole company. The same is with internet stores bringing the major income of the company.

In the development environment Phing helps to reduce the time required to preform frequently repeated actions. For example setting up the database requires a lot of clicking or in better cases write a shell command with a dozen of hard to remember parameters requiring user names and passwords.

Imagine the case when you’re developing your own search engine using his own index that needs to be rebuilt in order to include new possible search results. Debugging or making changes in the engine requires preforming the rebuilding process each time when you want to check if it’s working properly. This can be also related with removing loads of cache files etc. – that’s the task for Phing and its automation magic.

Creating a build target for the development environment could contain a operation filling the database with some test data which allows faster diagnosis of problems or investigating unwanted application behavior (reloading the whole application to the state preceding the controlled crash requires much less time and it’s always the same entry state).

The buildfile can also contain a set of requirement’s checks for the project. Before installing the app it’s worth checking if:

  • Given DB login credentials are valid
  • Directories used by the application are writable
  • A required server module / extensionis installed and enabled

Updating working application instances is always blood chilling. It’s mostly because of the requirement of data migration. The critical task while updating the application is to not lose any piece of that hardly gathered data. In this case a human error can be tragic in its consequences, thus minimizing the possibility of its occurrence is done by (for example.) checklists. An example checklist could look like the following:

  1. Switch the application into maintenance mode
  2. Create a database archive in a .tar.gz file and move it to the destination over SFTP: (destination)
  3. Create backup of the useruploads and log directories in a .tar.gzfile and place it on this FTP server (connection data)
  4. Preform a production database migration by executing listed SQL files in order that they appear on the list
  5. Preform a test database migration by executing listed SQL files in order that they appear on the list
  6. Execute scripts preforming integrity checks for the installation (list)
  7. Execute scrips preforming additional configuration steps in order defined below (list)
  8. Run the unit tests and make sure all of them are completed with success

Based on the list shown above, one can define proper targets with their dependencies and describe them using a buildfile called update.xml so it can be invoked using:

$ phing -f update.xml

One of pros of this solution is that the update scenario can be tested multiple times in a controlled development and test environments before moving to production.

Scope of automation applications

In general, using automation allows us to improve the quality of the development process and thus the final product. However the possibilities and scope of applications are restricted by independent factors such as the level of (de)organisation of work in the company.

Automation can be applied everywhere (as always it’s a matter of costs) – both in a small and chaotic company where programmer or a groups of programmers could aid the development process, or in a big, highly organised company for which the quality of the product is of vital importance.

It’s most effectively used in case of product that I call using the term “boxed” when all it’s required is to untar the package and run the build command in order to install it.

What’s worth automating?

Application configuration

Preparing configuration file templates for the application or its modules in function of the environment (development, test, production) containing different settings not only for the application itself but for the environment in general (memory limits, PHP error reporting, application log level, etc.) greatly speeds up deployment and adds a great possibility of fast reconfiguration.

Setting up permissions

This allows to eliminate the cases when the application fails to execute because of insufficient read/write/execute rights for a directory or a file. It’s mostly important on Unix systems. Automating these kind of operations speeds up both setting up a new working copy, or a production instance.

Application deployment / installation

Once the list of files to be put on the FTP server is created it reduces the number of missing files. Automation is also great when the application requires creation of additional files / directories, or needs to export some environmental variables.

Backups

Automating backups assures that the backup file is always properly named and stored in the right place – locally or on a remote server. Such targets could be scheduled using crontab.

Tool scripts

There is a class of scripts called tool scripts – small native scripts preforming a particular task such as loading the product catalog from a XML file, or setting up the application’s RBAC rules. There are cases when each of those scripts must be processed separately but it’s quite common that they have to be called in a previously defined order and with different (for that case) parameter values.

Complex and critical processes

Moving between software versions is quite unique for each pair of versions (updates / migrations). Thus they require a unique approach and scenario for each such passage. It’s recommended to create targets such as upgrade which can preform a proper requirement check in order to assure a valid upgrade process (module / application / library versions, backing up the database and current installation).

Working on with third party libraries

In case of more advanced projects many extensions or third party solutions are used. Those are ORM’s or search engines such as Lucene which have their own configuration files or a set of operations required to be preformed so the extension / library can work properly. It could rebuild some models based on the schema definition (Propel ORM) or build possible search results index in order to reduce the database load during search operations.

Helpers

Operations such as cleaning up temporary files, output cache, reloading services or building a tar archive of a selected revision.

About Grzegorz Godlewski

Senior Software Developer - Graduate of the Wroclaw University of Technology in Poland (Msc. Eng.). Currently working for SMT Software as a PHP / C# Backend Developer