Skip Navigation
Falvey Memorial Library
Advanced
You are exploring: Home > Blogs

VuFind: Moving from Laminas\Console to Symfony\Console

Background

When VuFind 2.0 came out in 2013, it marked a big step forward for VuFind, replacing a largely homegrown application design with Zend Framework 2. Using standard components clarified many aspects of VuFind’s design and removed some significant design limitations.

A lot has changed since 2013; the Composer tool has standardized the way PHP libraries and components are installed and shared, and a series of recommendations by the PHP-FIG has led to greater interoperability between established projects. As a result, once-monolithic projects like Zend Framework have broken into smaller parts, and it has become easier to pick and choose between components in order to find the best tools for solving particular problems.

The VuFind community is now in the process of developing VuFind 7.0, and one of the significant achievements of this release will be bringing VuFind’s dependencies up to date once again, to reflect the latest changes in the field. One big part of this was the recent renaming of Zend Framework to Laminas, though this change has proven to be more cosmetic than functionally significant. Another major development is the abandonment of the laminas-console (formerly zend-console) package; this has required significant work to accommodate, as will be detailed below.

Strengths and Weaknesses of laminas-console

When VuFind first adopted Zend Framework, it needed a method for developing command line utilities. At that point in time, there was really only one logical choice: the zend-console functionality built into the monolithic framework.

The zend-console approach to building command-line tools was designed to be similar to the framework’s general model-view-controller architecture. Commands were designed as actions within controller classes, and they could be run through the same public/index.php file that was also used for rendering web results — the code simply detected whether it was running in a command line or web context and ran appropriate code accordingly. This made it very easy to switch between developing web code and console code, and all of the standard VuFind resources could be accessed in exactly the same way.

The zend-console/laminas-console component also provided abstractions for input and output (allowing command line tools to behave the same across operating systems without requiring programmers to worry about the details) and used the framework’s routing logic to automate some of the work of processing command line arguments and options. All of this made code easier to read and write.

However, zend-console/laminas-console had some significant weaknesses, and these became more apparent as time went on. The use of controller classes led to unrelated actions being grouped together into the same files (because it’s easier to add an action to an existing controller than to build a whole new controller), resulting in bloated and hard-to-manage classes. The widespread use of static methods like Console::isConsole() and Console::write() made code difficult or impossible to test with PHPUnit, since using test doubles was not convenient. The command line argument/option processing lacked richness and did very little to enforce consistency of terminology between actions.

While we could certainly have continued to live with the drawbacks of laminas-console, the previously-mentioned developments in the PHP community mean that we no longer have to. Just as Zend Framework eventually evolved into a set of independent Laminas components, so did the competing Symfony framework split into smaller, reusable parts. While I continue to feel that Laminas is a good fit for VuFind’s overall use case, we are now in the position to take Symfony’s superior console support and incorporate that into VuFind without losing the other benefits of Laminas throughout the rest of the application. I applaud the Laminas development team for recognizing their own strengths and weaknesses, and making the wise decision to deprecate laminas-console; now, VuFind and other projects can take advantage of a more mature and feature-rich console component, and the Laminas team can focus more effort on their areas of strength.

Strengths of symfony-console

Symfony’s console component shares most of the existing strengths of laminas-console while also overcoming most of its weaknesses. It, too, provides abstractions of input and output and a mechanism for managing command line arguments and options… it just does it all a little bit better than laminas-console did. Rather than using a “controller and action” design, each Symfony console command is a single class, which makes the code organization cleaner and fits better with VuFind’s leaning toward explicit dependency-injection. Rather than relying on an external router configuration, the argument and option configuration is handled internally by the command classes. Argument and option configuration is also richer and more readable (if also a bit more verbose) than the laminas-console equivalent, with automatic, built-in support for consistent and attractive help screens. Symfony’s console component also ships with tools that make testing commands through PHPUnit extremely simple.

The bottom line: migrating from laminas-console to symfony-console offered an opportunity to not just update some underlying components but also to improve the design, user interface and test coverage of every command line tool in VuFind, all of which will contribute to easier maintenance and better user satisfaction in the future.

Migration Goals and Challenges

VuFind always aims to maintain backward compatibility to the greatest degree possible. Obviously, there is no way to avoid the fact that code implementing command line tools needs to be significantly rewritten… but just because code internals change, there is no reason for external interfaces to be redesigned; this migration should not, for example, break people’s existing cron jobs that depend on command line tools.

VuFind already supports two different methods for running some of its command-line tools: there is the “Laminas way” of calling public/index.php from the command line with controller and action parameters, but there are also some stand-alone PHP scripts (which are simply wrappers around public/index.php) that were created for backward-compatibility with tools developed as part of VuFind 1.x. The migration to Symfony console was designed with the goal of keeping all of these things working.

To make a long story short, the project was a success, and all of the work can be seen in pull request #1571. There were a few noteworthy challenges encountered along the way, however….

The first challenge was figuring out how to meld together the Symfony and Laminas ways of doing things. Since VuFind is largely Laminas-driven, we wanted to be sure that normal Laminas bootstrapping took place so that command line tools could be coded consistently with web-based tools, as in the past. However, Symfony offered some minor wrinkles that differed from past practice — most significantly, the fact that every command needed to have a single name, rather than the two-part controller and action convention used by laminas-console.

While it might have been possible to create a separate entry point to the Symfony console code, it seemed to make sense to continue to run command line tools through public/index.php as in the past. This not only helps with backward compatibility, but also allows general framework bootstrapping code to exist in just one place. The main difference is that, rather than being able to allow Laminas to internally detect console mode and behave differently, we instead need to run an entirely different bootstrap process in command line mode. The VuFindConsole\ConsoleRunner class was introduced to do this new work. It works together with VuFindConsole\Command\PluginManager to neatly fit Symfony console commands into the Laminas environment. The single-name vs. two-part name problem was solved by giving the Symfony commands slash-separated names; it’s illegal to put spaces in a Symfony command name, but slash is allowed. Thus, a “controller action”-style laminas-console command like “language normalize” becomes “language/normalize” in Symfony. VuFindConsole\Command\PluginManager is a standard Laminas plugin manager which allows Symfony commands to be built using standard Laminas factories, and which allows the commands to be looked up using their names as aliases. The VuFindConsole\ConsoleRunner does a bit of clever manipulation on the incoming command line parameters to normalize old-style “controller action” requests into new-style “controller/action” requests, then pulls the appropriate command out of the PluginManager and runs it. (If no command is specified, it loads ALL of the commands in the plugin manager so it can display a comprehensive help message). End result: seamless integration between Laminas and Symfony.

Once this framework was put in place, the rest of the work was simply a matter of converting controllers into command classes and writing tests (which brings the VuFindConsole module from having no test coverage at all, to having a very high degree of coverage). Everything that laminas-console could do, Symfony console could do as well or better. A few details of option processing occasionally required some searching and trial-and-error to figure out (for example, dealing with options that accept optional values was not immediately intuitive), but the documentation is good and the patterns are consistent, so the work went quite smoothly.

If there is any disadvantage at all to moving from controllers to stand-alone command classes, it is that shared convenience/utility methods are less readily available. However, this was really not much of a problem during the migration process. In the few cases where shared functionality was needed, it could be made available using either inheritance or traits. Many of the existing support methods in the controllers were used to fetch dependencies on the fly, a bad habit that was made unnecessary by switching to factory-driven dependency injection when refactoring the code.

As an added bonus, it was even possible to reimplement VuFind’s install.php script as a Symfony command, moving what was previously a completely stand-alone PHP script into a more object-oriented space (and bringing with it the benefits of testing and style normalization).

Migrating Custom Code

If you have built any custom command-line tools in your local module, you will have to migrate these to Symfony when you upgrade to VuFind 7.0. Fortunately, by looking at VuFind’s many existing commands and by reading Symfony’s excellent documentation, you should be able to do this with relative ease. By the time 7.0 is released, the plugin generator should be able to help you with the creation of new commands; watch pull request #1573 for progress on that detail. In the meantime, you can always reach out for support as needed.


Like
1 People Like This Post

Changing the Subject with VuFind

Background

Recent discussions related to the documentary, Changing the Subject, raised an interesting technical question: what should you do if your local needs come into conflict with national practices for describing a particular subject?

While Changing the Subject specifically addresses the conflict between using the terms “Illegal aliens” vs. “Undocumented immigrants,” we took this conversation as an opportunity to reduce user confusion over a whole host of terms using the word “Alien” to mean something other than “Extraterrestrial.”

The challenge, of course, is that this is not a problem that can be easily solved by editing records in our Integrated Library System. Not only are the tools for changing headings fairly difficult to use, but there is also the problem that new records will be constantly getting loaded into the system as we acquire new items, making maintenance an ongoing headache.

Enter VuFind: since we have an open source discovery layer to allow users to search our collection, and all of the records in our Integrated Library System are automatically loaded into VuFind through a software process, this gives us an opportunity to introduce some data transformations. By solving the problem once, we can introduce a system that will automatically keep the problem solved over time, without any ongoing record-editing maintenance.

Solution: Part 1 – Indexing Rules

The first part of the solution is to introduce some mapping into our MARC record indexing rules. We ended up adding these lines to our marc_local.properties file in VuFind’s local import directory:

topic_facet = 600x:610x:611x:630x:648x:650a:650x:651x:655x, (pattern_map.aliens)
topic = custom, getAllSubfields(600:610:611:630:650:653:656, " "), (pattern_map.aliens2)
pattern_map.aliens.pattern_0 = ^Alien criminal(.*)=>Noncitizen criminal$1
pattern_map.aliens.pattern_1 = ^Alien detention centers(.*)=>Detention centers$1
pattern_map.aliens.pattern_2 = ^Alien labor(.*)=>Noncitizen labor$1
pattern_map.aliens.pattern_3 = ^Alien property(.*)=>Foreign-owned property$1
pattern_map.aliens.pattern_4 = ^Aliens(.*)=>Noncitizens$1
pattern_map.aliens.pattern_5 = ^Children of alien laborers(.*)=>Children of noncitizen laborers$1
pattern_map.aliens.pattern_6 = ^Illegal alien children(.*)=>Undocumented immigrant children$1
pattern_map.aliens.pattern_7 = ^Illegal aliens(.*)=>Undocumented immigrants$1
pattern_map.aliens.pattern_8 = ^Children of illegal aliens(.*)=>Children of undocumented immigrants$1
pattern_map.aliens.pattern_9 = ^Women illegal aliens(.*)=>Women undocumented immigrants$1
pattern_map.aliens.pattern_10 = keepRaw
pattern_map.aliens2.pattern_0 = ^Alien criminal(.*)=>Noncitizen criminal$1
pattern_map.aliens2.pattern_1 = ^Alien detention centers(.*)=>Detention centers$1
pattern_map.aliens2.pattern_2 = ^Alien labor(.*)=>Noncitizen labor$1
pattern_map.aliens2.pattern_3 = ^Alien property(.*)=>Foreign-owned property$1
pattern_map.aliens2.pattern_4 = ^Aliens(.*)=>Noncitizens$1
pattern_map.aliens2.pattern_5 = ^Children of alien laborers(.*)=>Children of noncitizen laborers$1
pattern_map.aliens2.pattern_6 = ^Illegal alien children(.*)=>Undocumented immigrant children$1
pattern_map.aliens2.pattern_7 = ^Illegal aliens(.*)=>Undocumented immigrants$1
pattern_map.aliens2.pattern_8 = ^Children of illegal aliens(.*)=>Children of undocumented immigrants$1
pattern_map.aliens2.pattern_9 = ^Women illegal aliens(.*)=>Women undocumented immigrants$1
pattern_map.aliens2.pattern_10 = (.*)=>$1

This uses two very similar, but subtly different, pattern maps to translate terminology going to the topic_facet and topic fields.

The key difference is in pattern_10 of the two maps — for the “aliens” pattern, we use the “keepRaw” rule. This means that we translate headings when they match one of the preceding patterns, and we keep them in an unmodified form when they don’t. Thus, with this pattern, none of the headings will ever get indexed in their original forms; only in the translated versions. This is because we do not want outdated terminology to display in search facet lists.

On the other hand, in the “aliens2” pattern, we use a regular expression to always index the original terminology IN ADDITION TO any translated terminology. Even though we have opinions about which terms should be displayed, users may still have expectations about the older terminology. By indexing both versions of the terms, we make sure that searches will work correctly no matter how the user formulates their query.

Solution: Part 2 – Custom Code

Unfortunately, index rules alone do not fully solve this problem. This is because when working with MARC records, VuFind displays subject headings extracted directly from the raw MARC data instead of the reformatted values stored in the Solr index. This allows VuFind to take advantage of some of the richer markup found in the MARC, but in this situation, it means that we need to do some extra work to ensure that our records display the way we want them to.

The solution is to create a custom record driver in your local VuFind installation and override the getAllSubjectHeadings() method to do some translation equivalent to the mappings in the import rules. Here is an example of what this might look like:

<?php

namespace MyVuFind\RecordDriver;

class SolrMarc extends \VuFind\RecordDriver\SolrMarc
{
    /**
     * Translate "alien" headings.
     *
     * @param string $heading Input string
     *
     * @return string
     */
    protected function dealienize($heading)
    {
        static $regexes = [
            '/^Alien criminal(.*)/' => 'Noncitizen criminal$1',
            '/^Alien detention centers(.*)/' => 'Detention centers$1',
            '/^Alien labor(.*)/' => 'Noncitizen labor$1',
            '/^Alien property(.*)/' => 'Foreign-owned property$1',
            '/^Aliens(.*)/' => 'Noncitizens$1',
            '/^Children of alien laborers(.*)/' => 'Children of noncitizen laborers$1',
            '/^Children of illegal aliens(.*)/' => 'Children of undocumented immigrants$1',
            '/^Illegal alien children(.*)/' => 'Undocumented immigrant children$1',
            '/^Illegal aliens(.*)/' => 'Undocumented immigrants$1',
            '/^Women illegal aliens(.*)/' => 'Women undocumented immigrants$1',
        ];
        foreach ($regexes as $in => $out) {
            $heading = preg_replace($in, $out, $heading);
        }
        return $heading;
    }

    /**
     * Get all subject headings associated with this record.  Each heading is
     * returned as an array of chunks, increasing from least specific to most
     * specific.
     *
     * @param bool $extended Whether to return a keyed array with the following
     * keys:
     * - heading: the actual subject heading chunks
     * - type: heading type
     * - source: source vocabulary
     *
     * @return array
     */
    public function getAllSubjectHeadings($extended = false)
    {
        // Get extended headings from the parent:
        $headings = parent::getAllSubjectHeadings(true);

        foreach ($headings as $i => $heading) {
            if (isset($headings[$i]['heading'][0])) {
                $headings[$i]['heading'][0]
                    = $this->dealienize($headings[$i]['heading'][0]);
            }
        }

        // Reduce to non-extended format if necessary:
        if (!$extended) {
            $reduce = function ($var) {
                return serialize($var['heading']);
            };
            return array_map(
                'unserialize', array_unique(array_map($reduce, $headings))
            );
        }
        return $headings;
    }
}

You can learn more about building custom record drivers in the VuFind wiki.

Conclusions

While it requires a bit of redundancy, solving this problem with VuFind is still a great deal simpler and less painful than trying to maintain the records in a different way. If you would like to adopt a similar solution in your library and need more information beyond the code and configuration shared here, please feel free to reach out to the VuFind community through one of the methods listed on our support page.


Like
1 People Like This Post

Happy Hacktoberfest!

By Chris Hallberg, Library Technology Developer

Open-source software is an amazing movement in today’s programming environment. By sharing the code behind programs, open-source projects empower online communities to create quality programs that are available for free. These collaborations celebrate transparency and inclusion, improving the landscape of development in many ways. Many of the programs you likely use are developed by open-source communities. The Firefox and Chrome browsers, the Android operating system, and many websites are entirely or use open-source software.

If that sounds appealing to you, there is no better time to dip your feet into open-source development than now. Welcome to Hacktoberfest! Hacktoberfest is a time where the largest repository of open-source software, GitHub, encourages people to try out development and encourages its members and projects to make the barriers of entry as low as possible. Beginner projects are created for practice. Established projects tag issues that are good for new coders with a special Hacktoberfest label. Contribute enough code in the month of October and you will be sent a free shirt!

 

The shirts are awesome!

 

So how do you get started? The first thing you have to do is learn a special tool called “version control”. If you’ve ever been working on a paper that requires revisions, you have experience with version control techniques.

 

 

What version control tools do is allow you to do is to bookmark changes and stages of progress without making a hundred different files. They also make it easy to go back to previous versions and see the differences between bookmarks. One of the most popular options is called git (hence GitHub).

The most important thing that git allows you to do is to reconcile different changes to the same files. You can “merge” changes from one version into another. This allows multiple people to work on the same files without conflicting with each others’ work. This means that teams and communities can delegate tasks and work on them individually. GitHub gives them a place to host code publically and git allows people to take projects in their own directions (called “forks”).

 

When someone wants to add their improvements back to the project, they create what’s called a “pull request”. A pull request shows all of the changes that have been made to a project clearly and asks the original owner if they want to include these changes in the project. It’s the center for conversation and progress on GitHub.

Now that I’ve covered the basics, it’s time for you to dive in and make your own pull requests! Creating only four scores you an awesome shirt. Go to the Hacktoberfest website to get started or check out this resource or this interactive tutorial for git to start applying that to your everyday routine.


Chris Hallberg is a web designer and technology developer at Falvey Library, an open-source enthusiast and part-time teacher.


Like
1 People Like This Post

Happy Hacktoberfest!

Open-source software is an amazing movement in today’s programming environment. By sharing the code behind programs, open-source projects empower online communities to create quality programs that are available for free. These collaborations celebrate transparency and inclusion, improving the landscape of development in many ways. Many of the programs you likely use are developed by open-source communities. The Firefox and Chrome browsers, the Android operating system, and many websites are entirely or use open-source software.

If that sounds appealing to you, there is no better time to dip your feet into open-source development than now. Welcome to Hacktoberfest! Hacktoberfest is a time where the largest repository of open-source software, GitHub, encourages people to try out development and encourages its members and projects to make the barriers of entry as low as possible. Beginner projects are created for practice. Established projects tag issues that are good for new coders with a special Hacktoberfest label. Contribute enough code in the month of October and you will be sent a free shirt!

 


The shirts are awesome

 

So how do you get started? The first thing you have to do is learn a special tool called “version control”. If you’ve ever been working on a paper that requires revisions, you have experience with version control techniques.

 

 

What version control tools do is allow you to do is to bookmark changes and stages of progress without making a hundred different files. They also make it easy to go back to previous versions and see the differences between bookmarks. One of the most popular options is called git (hence GitHub).

The most important thing that git allows you to do is to reconcile different changes to the same files. You can “merge” changes from one version into another. This allows multiple people to work on the same files without conflicting with each others’ work. This means that teams and communities can delegate tasks and work on them individually. GitHub gives them a place to host code publically and git allows people to take projects in their own directions (called “forks”).

 

When someone wants to add their improvements back to the project, they create what’s called a “pull request”. A pull request shows all of the changes that have been made to a project clearly and asks the original owner if they want to include these changes in the project. It’s the center for conversation and progress on GitHub.

Now that I’ve covered the basics, it’s time for you to dive in and make your own pull requests! Creating only five scores you an awesome shirt. Go to the Hacktoberfest website to get started or check out this resource or this interactive tutorial for git to start applying that to your everyday routine.

 


 

Chris Hallberg is a web designer and technology developer at Falvey Library, an open-source enthusiast and part-time teacher.


Like

Automatically updating locally customized files with Git and diff3

The Problem

VuFind follows a fairly common software design pattern: it discourages users from making changes to core files, and instead encourages them to copy files out of the core and modify them in a local location. This has several advantages, including putting all of your changes in one place (very useful when a newcomer needs to learn how you have customized a project) and easing upgrades (you can update the core files without worrying about which ones you have changed).

There is one significant disadvantage, however: when the core files change, your copies get out of sync. Keeping your local copies synched up with the core files requires a lot of time-consuming, error-prone manual effort.

Or does it?

The Solution

One argument against modifying files in a local directory is that, if you use a version control tool like Git, the advantages of the “local customization directory” approach are diminished, since Git provides a different mechanism for reviewing all local changes to a code base and for handling software updates. If you modify files in place, then “git merge” will help you deal with updates to the core code.

Of course, the Git solution has its own drawbacks — and VuFind would lose some key functionality (the ability for a single instance to manage multiple configurations at different URLs) if we threw away our separation of local settings from core code.

Fortunately, you can have the best of both worlds. It’s just a matter of wrangling Git and a 3-way merge tool properly.

Three Way Merges

To understand the solution to the problem, you need to understand what a three-way merge is. Essentially, this is an algorithm that takes three files: an “old” file, and two “new” files that each have applied different changes to the “old” file. The algorithm attempts to reconcile the changes in both of the “new” files so that they can be combined into a single output. In cases where each “new” file has made a different change in the same place, the algorithm inserts “conflict markers” so that a human can manually reconcile the situation.

Whenever you merge a branch in Git, it is doing a three-way merge. The “old” file is the nearest common ancestor version between your branch and the branch being merged in. The “new” files are the versions of the same file at the tips of the two branches.

If we could just do a custom three-way merge, where the “old” file was the common ancestor between our local version of the file and the core version of the file, with the local/core versions as the “new” files, then we could automate much of the work of updating our local files.

Fortunately, we can.

Lining Up the Pieces

Solving this problem assumes a particular environment (which happens to be the environment we use at Villanova to manage our VuFind instances): a Git repository forked from the main VuFind public repository, with a custom theme and a local settings directory added.

Assume that we have this repository in a state where all of our local files are perfectly synched up with the core files, but that the upstream public repository has changed. Here’s what we need to do:

1.) Merge the upstream master code so that the core files are updated.

2.) For each of our locally customized files, perform a three-way merge. The old file is the core file prior to the merge; the new files are the core file after the merge and the local file.

3.) Manually resolve any conflicts caused by the merging, and commit the local changes.

Obviously step 2 is the hard part… but it’s not actually that hard. If you do the local updates immediately after the merge commit, you can easily retrieve pre-merge versions of files using the “git show HEAD~1:/path/to/file” command. That means you have ready access to all three pieces you need for three-way merging, and the rest is just a matter of automation.

The Script

The following Bash script is the one we use for updating our local instance of VuFind. The key piece is the merge_directory function definition, which accepts a local directory and the core equivalent as parameters. We use this to sync up various configuration files, Javascript code and templates. Note that for configurations, we merge local directories with core directories; for themes, we merge custom themes with their parents.

The actual logic is surprisingly simple. We use recursion to navigate through the local directory and look at all of the local files. For each file, we use string manipulation to figure out what the core version should be called. If the core version exists, we use the previously-mentioned Git magic to pull the old version into the /tmp directory. Then we use the diff3 three-way merge tool to do the heavy lifting, overwriting the local file with the new merged version. We echo out a few helpful messages along the way so users are aware of conflicts and skipped files.

#!/bin/bash

function merge_directory
{
    echo merge_directory $1 $2
    local localDir=$1
    local localDirLength=${#localDir}
    local coreDir=$2

    for current in $localDir/*
    do
        local coreEquivalent=$coreDir${current:$localDirLength}
        if [ -d "$current" ]
        then
          merge_directory "$current" "$coreEquivalent"
        else
          local oldFile="/tmp/tmp-merge-old-`basename "$coreEquivalent"`"
          local newFile="/tmp/tmp-merge-new-`basename "$coreEquivalent"`"
          if [ -f "$coreEquivalent" ]
          then
            git show HEAD~1:$coreEquivalent > $oldFile
            diff3 -m "$current" "$oldFile" "$coreEquivalent" > "$newFile"
            if [ $? == 1 ]
            then
              echo "CONFLICT: $current"
            fi
            cp $newFile $current
          else
            echo "Skipping $current; no equivalent in core code."
          fi
        fi
    done
}

merge_directory local/harvest harvest
merge_directory local/import import
merge_directory local/config/vufind config/vufind
merge_directory themes/vuboot3/templates themes/bootstrap3/templates
merge_directory themes/villanova_mobile/templates themes/jquerymobile/templates
merge_directory themes/vuboot3/js themes/bootstrap3/js
merge_directory themes/villanova_mobile/js themes/jquerymobile/js

Conclusion

I’ve been frustrated by this problem for years, and yet the solution is surprisingly simple — I’m glad it finally came to me. Please feel free to use this for your own purposes, and let me know if you have any questions or problems!


Like

VuFind: The Year in Review, 2013

In October 2012, Falvey Memorial Library hosted a well-attended meeting of software developers interested in VuFind, the open source search software developed at Falvey and used for the local library catalog and Digital Library. A year later, the 2013 VuFind Summit is about to take place, providing a good opportunity to reflect on VuFind’s progress in recent months.

One highlight of the 2012 Summit was a preview of VuFind 2.0, a completely rewritten update making the software significantly more flexible and easier to use. Following the Summit, several early adopters began using VuFind 2.0 even before it was officially released. It eventually came out, right on schedule, in June of this year. A 2.1 release in August, added still more features, including the ability to search websites (a feature that powers the search of this very blog). Work has already begun on VuFind 2.2, and there is no sign of an end to its continuing growth and improvement.

Beyond Villanova, others are also enthusiastically discussing the software. September 2013 saw a large group of VuFind users meet up in Germany, and VuFind was also prominently featured in a major Italian conference focusing on the use of open source software in libraries.

We are very proud of the success of VuFind and look forward to a continued collaboration with the incredibly diverse community that uses the software. To learn more, visit http://vufind.org.

VuFind TUHH installation

Hamburg University of Technology

VuFind Donostia installation

Library Network of Saint Sebastian

 

VuFind Ireland installation

National Library of Ireland

 


Demian Katz is a Library Technology Development Specialist and VuFind Developer.


Like

A More Complex VuFind 2 Unit Test

Okay, much sooner than expected, I found a good candidate for some more complex unit testing: the VuFindCart object that we mocked in the previous post.

Preparing for Testing

As I mentioned in the previous post, sometimes a bit of work needs to be done before classes can be easily tested.

In the case of VuFindCart, the class reads and writes cookie values (since this is how VuFind tracks book bag contents).  This is problematic for testing — the class is dependent on web-specific elements in the environment.  The solution is easy enough, though: add some extra abstraction.

First, we add the ability to override $_COOKIE through an extra constructor parameter: diff.

Next, we create a wrapper function around setcookie(): diff.

Now we can test everything easily even if we don’t have real cookies.  This abstraction may also come in handy in the future if Zend Framework offers its own cookie handling utilities, since we have reduced the number of times that VuFind directly utilizes low-level PHP functionality.

Writing the Test

As of this writing, we don’t have 100% test coverage for VuFindCart, but the work-in-progress test demonstrates a few useful techniques.  You might want to open the full code in another tab so you can look at it as you read the notes below.

First of all, as discussed in the previous post, VuFindCart depends on VuFindRecordLoader.  I use a mock object to satisfy that dependency.  The mock object is created as a property of the test class — this way, if tests need to make assertions about how VuFindCart interacts with VuFindRecordLoader, these can easily be set up.  (I haven’t done this yet, but it will be necessary to get 100% coverage).

I have created a getCart() convenience method which returns a VuFindCart mock object.  This routine uses the second parameter of getMock(), which can be used to specify which methods are to be mocked out.  In this case, we are only mocking out the setCookie() wrapper function, since we don’t want our test to actually call PHP’s setcookie() function.  Everything else will be executing real code from the VuFindCart class.

Most of the actual test cases are simple low-hanging fruit — setting values in the constructor and making sure that the corresponding get methods return them correctly, etc.  It should be fairly self explanatory.

The testFullCart() method might be of some interest — this demonstrates using a series of assertions to confirm that an expected sequence of events happens in the appropriate order (in this case, we are making sure that the cart registers as being full at the right point in time).

The testCookieWrite() method further demonstrates the power of mock objects.  We want to test that adding an item to the cart writes the correct values to the cookie.  Since we have mocked out the setCookie() method, we can set up expectations that it will be called at particular times with particular values — this allows us to test that a call to $cart->addItem() results in the expected calls to $cart->setCookie(), without actually writing any real cookies.  The PHPUnit syntax can seem a bit weird at first, but it’s not really too bad.  In this example, $this->at(0) and $this->at(1) are being used to set up rules relating to the first and second calls to $cart->setCookie; the ->with() portion sets up expectations for particular parameters at each call.

Conclusions

I hope this further demonstrates how testing can be achieved.  As before, please feel free to ask questions!


Like
1 People Like This Post

A Simple VuFind 2 Unit Test

I have said on several occasions that test-writing might eventually become a good way for new developers to contribute to the VuFind project. By increasing our test coverage (and thus future code stability), new tests are a valuable contribution to the software… and the act of studying all the code paths in order to write a test is a good way to learn how things work.

Of course, the reality of the situation is not so straightforward. There is still a lot of code in VuFind that is inherently hard to test, usually due to complex dependencies. At this point in time, the process of writing tests is often also the process of refactoring code to reduce coupling, which in turn makes it more testable — and this often requires a deeper understanding of the system than a newcomer is likely to have. (See my previous blog post for an example of the sort of refactoring I mean).

So testing isn’t always easy…  but still, it can sometimes be straightforward.  Here’s a relatively simple example to demonstrate some of the principles.

Getting Set Up

This post assumes you have a clean copy of VuFind 2 checked out somewhere separate from your production instance, and that you have the necessary tools (PHPUnit, Phing, etc.) installed.  More background on the setup can be found in the VuFind wiki.

What We Are Testing

For this example, we are testing VuFind’s Cart view helper (VuFindViewHelperRootCart).  This is a very trivial piece of glue code whose purpose is to make VuFind’s cart object (which keeps track of user selections in the optional “book bag” feature) available for use in view scripts.  You don’t really gain much by writing a test for this class — it is unlikely to change much and it contains no complex logic — but since it is so simple, it’s a good candidate for this tutorial.  We can get to 100% coverage very quickly.

You can view the full code of the view helper in our Git repository.

There are only two methods: the constructor (which takes a VuFindCart object as a mandatory parameter) and __invoke (a PHP magic method which returns the Cart object when the helper is invoked as if it were a function).

Getting Started

By convention, unit tests (i.e. all tests that exercise VuFind functionality without relying on an active test instance) reside in module/VuFind/tests/unit-tests/src.  Within this directory, test classes are arranged and namespaced to correspond with the code that they test.

This means that we’re going to create a module/VuFind/tests/unit-tests/src/View/Helper/Root/CartTest.php, living in the VuFindTestViewHelperRoot namespace.

All PHPUnit test cases must be subclasses of the PHPUnit_Framework_TestCase class.  VuFind includes some base test classes with additional convenience methods (see the VuFindTest namespace), but this particular test is simple enough that we can simply extend the default base class:

class CartTest extends PHPUnit_Framework_TestCase

Writing the Test

One of the keys to writing an effective test is to avoid doing any work that is not related to the task at hand.  You don’t want your test to fail because of a problem in a different area of the code — this will make bugs harder to locate when something breaks.

We are trying to test something very simple here — essentially we want to be sure that when we put a cart object into the view helper’s constructor, we get the same object out when we invoke the helper.

This is where the “don’t do unrelated work” rule comes in.  We could construct a real VuFindCart object for testing purposes, but then if there was a bug in the VuFindCart constructor, that might cause our test to fail, even though that has nothing at all to do with our view helper.  Fortunately, we don’t have to construct a real VuFindCart object, thanks to PHPUnit’s mock object feature.

With mock objects, we can create objects that serve as placeholders for real classes in our code.  They accept the same method calls and pass the same instanceof tests as real objects, but they don’t actually do anything — unless we configure them to expect particular incoming data or simulate specific responses under specific circumstances.  These are a very valuable testing tool!

In our particular case, we don’t need to do anything fancy with mocks — we just need to call PHPUnit’s built-in $this->getMock() method to construct a fake Cart object.

There’s just one small issue.  If you just call:

$this->getMock(‘VuFindCart’);

You will get an error.  VuFindCart’s constructor expects a VuFindRecordLoader object, and we have to satisfy this dependency even when building a mock.  Fortunately, the third parameter of getMock() accepts constructor parameters for the new mock object, and nothing stops us from creating a mock VuFindRecordLoader to satisfy the dependency.  Thus, we end up with:

$cart = $this->getMock(
‘VuFindCart’, null, array($this->getMock(‘VuFindRecordLoader’))
);

Now that we have our fake cart, the rest is simple…  Just construct a view helper:

$helper = new VuFindViewHelperRootCart($cart);

…and then test that invoke works by making an assertion that invoking the helper will return the same object that we passed to the constructor:

$this->assertEquals($cart, $helper());

Assertions are the most important part of any test — these are what determine whether each test passes or fails.  Never write a test without any assertions!  PHPUnit includes a wide range of assertion methods, allowing you to express many different conditions.

Running the Test

Now that the code is written (full test class available here), it’s just a matter of using your VuFind test instance to run it, as described in the wiki.

After confirming that a new test passes on my local system, I push it to the Git master and then check the code coverage report in Jenkins after everything rebuilds.  In this case, I’m now seeing 100% coverage for the cart helper.

What Next?

I hope this has served as a helpful introduction to some fundamentals, but I realize that most real-life testing is significantly more complicated.  I may try to write an article describing a more difficult test in the future if time permits.  In the meantime, if you want to try your hand at test-writing, feel free to send me questions — I’ll be happy to recommend some areas that might be worth looking at, and I can help with any refactoring that may be necessary.


Like
1 People Like This Post

VuFind, Zend Framework 2, and Flash Messages

One minor weakness of VuFind 1.x is that it doesn’t always provide good feedback for user actions.  One of the advantages to switching to a standard framework is that we can leverage an existing “flash message” mechanism to display messages to users in a consistent way.  Zend Framework 2 includes a simple FlashMessenger helper, but it took a bit of learning to use it as effectively as possible.  I hope that a discussion of my solution to this problem will helpfully demonstrate some aspects of the framework and VuFind 2.x’s architecture.

The Problem

In Zend Framework, flash messages are stored in an object by the controller.  For example:

$this->flashMessenger()->setNamespace('info')
    ->addMessage('Email sent successfully.');

or

$this->flashMessenger()->setNamespace('error')
    ->addMessage('Operation failed.');

 

The view is then responsible for pulling information out of the object and formatting it for display.

I wanted VuFind to display messages in a consistent way, so it seemed to make the most sense to use a View Helper to render flash messages.  This helper could take care of pulling messages from the namespaces in the object, applying translation, and building <div> tags with appropriate classes. This way, all I would have to do is add this code to a template:

<?=$this->flashMessages()?>

 

and my flash messages would display.

There’s just one hard part: getting the object containing the messages from the controller to the view with a minimum of fuss.

Solution #1: Brute Force

My first solution was not hard to come up with, but I was never happy with it.

Controllers in Zend Framework 2 pass values to their corresponding views by returning a value (often a ZendViewModelViewModel object).  If we want to ensure that all of these objects contain a particular value, we can establish a convention where VuFind’s controllers never construct a ViewModel directly; instead, they always call a $this->createViewModel() method defined in an abstract base class.  This helper method is responsible for constructing a ViewModel and then attaching the flash messenger object to it.  Once the ViewModel has a flash messenger attached, the View Helper responsible for rendering the messages can simply pull the messenger from a known named property in the model.

Advantage: It works.

Disadvantage: It’s ugly.  It relies on a particular arbitrary value being in a particular arbitrary place.  It takes away some of the flexibility of the framework by forcing the user to always return ViewModels, and to construct them in a particular way.  It makes the user jump through a hoop (calling createViewModel to create a view model) and punishes them in an unintuitive way if they don’t comply (flash messages stop appearing).

Solution #2: Working with the Framework

Fortunately, there is a better way.

One of the nice features of ZF2 is its service manager system (described in more detail in this blog post).  Service managers are responsible for constructing and storing instances of important objects used by the application.  Ugly application-specific details can be pushed into service manager configuration so that the actual components of the application can be simpler and more reusable.

The flash messenger controller helper and our custom view helper are both constructed by service managers.  This means that, through proper service manager configuration, we can ensure that the flash messages view helper always has access to the flash messenger controller helper, with no need for the intervention of a nasty createViewModel() method.

This is achieved with the help of a factory in our view helper service manager configuration:

'flashmessages' => function ($sm) {
    $messenger = $sm->getServiceLocator()
        ->get('ControllerPluginManager')
        ->get('FlashMessenger');
    return new VuFindViewHelperRootFlashmessages(
        $messenger
    );
},

 

Here’s how it works: service manager factories are passed an instance of the service manager that is doing the object construction ($sm in the example above — an instance of the service manager devoted to view helpers). $sm->getServiceLocator() retrieves the service manager that constructed $sm — the top-level service manager that manages the whole VuFind application. From here, we can call ->get(‘ControllerPluginManager’) to obtain the service manager for controller plugins. We can then call ->get(‘FlashMessenger’) on the controller plugin service manager to get hold of the flash messenger. This object ($messenger) is then passed to the constructor of the new VuFindViewHelperRootFlashmessages helper. The object now has access to the flash messenger as soon as it is constructed. None of the other code needs to know about the details — it just works like magic.

Yes, this is a bit ugly. But the important thing is that the ugliness is now in the configuration of the application — part of the glue that holds everything together. It’s okay for this part of the code to be a bit convoluted and application-specific, because its whole purpose is to make the various parts of the application fit together. The beautiful thing is that those component parts don’t have to know anything about the details of how the glue works — as a whole, the application has become less coupled and more flexible. Compare this to solution #1, where the ugliness was more deeply embedded in the application’s controllers and in the behavior of the view helper itself.

The Code

If you want to see how VuFind transitioned from Solution #1 to Solution #2, take a look at these diffs.

The Bottom Line

There’s definitely a learning curve involved with ZF2, but I continue to feel that it is worth the effort.  I hope this example has shed a little light on some of the issues… but if anything is unclear (and I’m sure something is), feel free to ask questions.  I’m always happy to help, especially if it means that more people are potentially able to help with the process of improving the code.


Like

Moving VuFind to Zend Framework 2: Part 5 — The Dreaded forward() Bug

IMPORTANT UPDATE: The problem discussed in this post only applies to Zend Framework 2.0 RC4 and earlier. Starting with RC5, a fix was introduced which eliminates the double-rendering issue described below. With the fix in place, you can do the logical thing of returning the output of $this->forward()->dispatch() rather than having to return false. Thanks to Nicholas Calugar, Matthew Weier O’Phinney, and all the others on zf-contributors and #zftalk.2 who talked to me about this problem and helped me refine my pull request to fix it.


It’s a law of software development that if you can’t explain some weird behavior and you choose to ignore it, it will inevitably come back to bite you. I know this very well, but knowing it doesn’t prevent it from happening.

The Problem

When I first implemented VuFind’s Flash Messenger view helper (designed to display status messages stored in the session), I ran into a strange problem. The behavior of the helper was simple: check the session for messages, display them, then clear the session. The problem was that flash messages never displayed. Eventually I just commented out the “clear the session” line and everything started working. I didn’t have time to figure out what was really going on, so I just put a note in the code to check it later, and moved on to other things.

The Real Problem

I kept running into bizarre flash messenger behavior, so I decided I needed to solve this problem for real rather than just putting on a band-aid. After digging deep into ZF2 and putting logging lines all over the place, I figured out what was causing the problem: many of my view templates were getting rendered twice. Even though the templates displayed on screen just once, the code was executing twice. This was the reason for my flash messenger problem: the first render displayed the messages and cleared them. The second render replaced the first, and by this point, the messages were gone.

The Root Cause

After a lot more digging and some extremely valuable help from mpinkston on the #zftalk.2 IRC channel, I discovered the root cause. Sometimes you want to forward from one controller action to another. In ZF1, you could use “return $this->_forward(…)” to achieve the effect. In ZF2, there is a controller plug-in called forward() that does the same basic thing. I was using it like this:


return $this->forward()->dispatch('controllerName', array('action' => 'actionName'));

Seems harmless enough, but what actually happens here is that the forward() plug-in dispatches an action, which causes ZF2 to attach a view model to the top-level layout object. The forward() plug-in then returns that view model. When the calling controller returns it, ZF2 attaches a second copy to the layout object. And then both copies get rendered!

What I should have been doing was this:


$this->forward()->dispatch('controllerName', array('action' => 'actionName'));
return false;

A simple, honest mistake with evil, subtle consequences.

The Solution

I’ve already found a solution to my immediate problem, as described above. I won’t make this mistake again, and VuFind’s flash messenger now works correctly. I was concerned that the solution would only work when forwarding to actions that return a ViewModel, but it seems to work for all cases, even when a Response is returned instead (i.e. when redirecting to another page).

I remain concerned that this is a big potential pitfall for users of ZF2 in general. It’s an easy mistake to make, and the consequences are extremely non-obvious, especially since in many cases, the only result of the mistake is a slight performance hit caused by duplicate work being done.

Hopefully this will be addressed in the future either by notes in the documentation or by the introduction of a convenience method in the forward() plugin that returns false instead of the actual output of the dispatched action. If either of these things had existed, I probably wouldn’t have spent the better part of a work day chasing this problem. But I don’t mean to criticize the ZF2 team — I like the framework, and these are the perils of working with software that is still under development. I just point this out as something to watch out for that should be addressed, one way or another, before the final 2.0 official release.


Like

Next Page »

 


Last Modified: August 13, 2012