Parsing xml with Boost

Everyone would agree that xml is a rather clumsy format for transferring data, however one essential thing about xml is that it has become a de facto standard for passing information, and lots of platforms and libraries provide tools for parsing the format. One of such tools is also available in Boost libraries. It is easy to overlook, because it is not advertised under name “xml”, therefore I thought it deserves a post in my blog.

The library that we will need is Boost.PropertyTree, developed by Marcin Kalicinski. Its main focus is not parsing xml, but an observation that many data formats share the same general tree structure, that can conceptually, quoting the description of the library author, be described as follows.

struct ptree
{
   string data;                          // data associated with the node
   list< pair<string, ptree> > children; // ordered list of named children
};

Xml just happens to fall into this category of data structures. Boost.PropertyTree allows building data structure ptree from an xml file; then you can manually inspect the structure and collect the information you need. The library also allows building and modifying ptree and storing it into an xml file. Technically, the library is not 100% compatible with W3C specifications, but it is sufficient to correctly process data stored in form of xml file. While the library itself is very good, it doesn’t fully document its capabilities, so we will try to see by an example how such xml processing can be performed.

Reading xml

First, we will want to parse the following document.

<?xml version="1.0"?>
<sked>
  <version>2</version>
  <flight>
    <carrier>BA</carrier>
    <number>4001</number>
    <date>2011-07-21</date>
  </flight>
  <flight cancelled="true">
    <carrier>BA</carrier>
    <number>4002</number>
    <date>2011-07-21</date>
  </flight>
</sked>

The goal is to put the information from the file into the following data structure Sked.

typedef boost::gregorian::date Date;

struct Flight
{
    std::string  carrier;
    unsigned     number;
    Date         date;
    bool         cancelled;
};

typedef std::vector<Flight> Sked;

We start with the following code. It is explained in detail further below.

#include <boost/property_tree/xml_parser.hpp>
#include <boost/property_tree/ptree.hpp>

Sked read( std::istream & is )
{
    // populate tree structure pt
    using boost::property_tree::ptree;
    ptree pt;
    read_xml(is, pt);

    // traverse pt
    Sked ans;
    BOOST_FOREACH( ptree::value_type const& v, pt.get_child("sked") ) {
        if( v.first == "flight" ) {
            Flight f;
            f.carrier = v.second.get<std::string>("carrier");
            f.number = v.second.get<unsigned>("number");
            f.date = v.second.get<Date>("date");
            f.cancelled = v.second.get("<xmlattr>.cancelled", false);
            ans.push_back(f);
        }
    }

    return ans;
}

We did not include all the necessary header files, only those dealing with property tree. First, we create an empty ptree and populate it using function read_xml. It is defined in namespace boost::property_tree, but due to argument dependent look-up it can be used without any namespace qualification. We feed our function with std::istream rather than std::string or std::ifstream, because it is more generic an interface: it can accommodate both strings (in case we get the xml from some networking library) and files.

Once the reading of xml file is done we can begin querying pt for contents. We get the top-level child sked and iterate over its children. We pick only those with key flight, thus skipping the child version. We read data using function template get; we also specify the type we want to cast our data to. PropertyTree uses streaming operators << and >> to convert any type to and from string, much like boost::lexical_cast. In case the data element we read is optional, we can use get_optional which will return boost::optional rather than our target type directly. In case of attributes, it gets a bit more complicated. Structure ptree does not have a concept of an attribute, so xml attributes are just sub-elements of a special element <xmlattr>. For our attribute cancelled we use a third way of getting the data: we provide a default value that should be returned: false. this is similar to boost::optional’s get_value_or. Note that we didn’t have to provide the template parameter, because it could be deduced from the default value.

PropertyTree customizations

The above example has one significant fault. If we try to parse the xml file we listed above, our function fails, and throws an exception. This is because type Date, although it provides streaming operators, uses different string formats than xml file. Default string format for Date is "2011-Jul-30". What we need it extended ISO date format: "2011-07-30". If in a hurry, we could solve the problem by setting the global locale object to the one that uses our date format. However, this may spoil other parts of our program that rely on different date format. We need a more localized solution that will change the date format only for xml parser. Boost.Property allows easy per-type customizations. We will need to define the following type.

class DateTranslator 
{
    std::locale locale_; // implementation detail

public:
    typedef std::string  internal_type;
    typedef Date         external_type;

    DateTranslator();
    boost::optional<external_type> get_value( internal_type const &v );
    boost::optional<internal_type> put_value( external_type const& v );
};

Our ‘translator’ is capable of encoding Date into std::string and vice-versa. boost::optional in return value is used for signalling conversion errors. You need to define two types internal_type and external_type, even if they are spelled longer than the original types. It is based on those two types that the framework determines (using boost::enable_if) that our class is a translator. Now, we need to define the two conversions. It is up to you how you want to implement the conversions. Here, we chose to do that by using a temporary stream for which we will customize the locale. First a function that returns the desired locale:

std::locale isoDateLocale() {
    typedef boost::date_time::date_facet<Date, char> tOFacet;
    typedef boost::date_time::date_input_facet<Date, char> tIFacet;

    std::locale loc;
    loc = std::locale( loc, new tIFacet("%Y-%m-%d") );
    loc = std::locale( loc, new tOFacet("%Y-%m-%d") );
    return loc;
}

We need to add two facets to our locale: one for reading dates, the other for writing. First we default-create a locale; it is a copy of the global locale. Then, we add the two facets that come with Boost.Date_Time library. In the default constructor we use this function to initialize our private object locale_. Next, we define the two conversion functions by creating a string-based stream, imbuing our locale, and just doing the regular streaming:

boost::optional<DateTranslator::external_type> 
DateTranslator::get_value(internal_type const& v)
{ 
    std::istringstream stream(v);
    stream.imbue(locale_);
    external_type vAns;
    if( stream >> vAns ) {
        return vAns;
    }
    else {
        return boost::none;
    }
}

boost::optional<DateTranslator::internal_type> 
DateTranslator::put_value(external_type const& v)
{ 
    std::ostringstream ans;
    ans.imbue(locale_);
    ans << v;
    return ans.str();
}

Now, in order to use our new translator in PropertyTree we have two options. Either we create this object and pass it to function get as an additional parameter:

DateTranslator tr;
f.date = v.second.get<Date>( "date", tr );

Or, we can teach the framework that it should always use our translator internally without providing the additional argument each time. We do it by specializing class template translator_between:

namespace boost{ namespace property_tree{
    template<> 
    struct translator_between<std::string, Date>
    {
        typedef DateTranslator type;
    };
}}

Now our original call will work correctly:

f.date = v.second.get<Date>("date");

Generating xml

Now, for the last step, we will write a function that given the schedule of type Sked, will write an xml file into a stream.

void write( Sked sked, std::ostream & os )
{
    using boost::property_tree::ptree;
    ptree pt;

    pt.add("sked.version", 3);

    BOOST_FOREACH( Flight f, sked ) {
        ptree & node = pt.add("sked.flight", "");

        node.put("carrier", f.carrier);
        node.put("number", f.number);
        node.put("date", f.date);
        if( f.cancelled ) node.put("<xmlattr>.cancelled", true);
    }

    write_xml( os, pt );
}

The full working example of the xml parser is here. Or if you prefer a PDF file: parse_xml.pdf.