xforms

Is XForms really MVC?

Submitted by administrator on Fri, 09/03/2010 - 03:56.

This epic posting on MVC helped me better understand the pattern, and all the variants that have flowed outward from the original design. One interesting observation is that the earlier designs used Views primarily as output-only, and Controllers primarily as input-only, and as a consequence the Controller was the one true path for getting data [...]

The Illusion of a Web SOA distinct from Enterprise SOA

Submitted by administrator on Fri, 05/21/2010 - 23:53.

In "enterprise" service-oriented architectures (SOAs), XML is a clearly entrenched format for server-to-server communications. But there's a movement afoot to define a "Web SOA" as a separate entity that describes server-to-client communications... based on JSON as the format.

The rationale is that JSON parses "a hundred times faster" in a web browser than does XML. This is like the tail wagging the dog. It is a myopic, accept-the-status-quo approach to web application computing, and it just won't do.

First of all, the difference between parsing different serializations is negligible (when done right), and nodes are nodes are nodes, so if there is a 100-fold difference, it is because one parsing method is direct and the other is doing something phenomenally suboptimal and probably easy to fix. One way to solve this is to alter all the system architectures on the web and introduce the complexity of a data format transformation to cross the illusory chasm between a "Web SOA" and the "Enterprise SOA". Another idea would be to put the pressure where it rightly belongs, on web browser makers to fix the feature of efficient XML processing.

Particularly as vertical industry standardization continues, such as is happening in both insurance and healthcare, this latter solution is really the only viable one in the long term. XForms technologies like Lotus Forms and Ubiquity XForms wrap a logical client application around the exact XML that is needed by the server side because this is the best way to minimize system development and maintenance costs. Once there is a defined XML data model, there should not be some other second data model introduced into web applications for some reason as vapid as poor implementation of the required feature.

This conversion is not a trifling matter either; have you seen how ugly the JSON is that is actually capable of preserving XML fidelity (i.e. supporting round-tripping)? Trying to build a reasonable client-side around that JSON is not nearly such a pretty sight as when the data representational limitations of JSON are accepted in an application. If the full representational power of XML schema is used to define an application data model, then we need to be able to carry that out to the client without significant alteration so that we can make references to the data transparently, not through the lens of some horrendous transformation. And make no mistake, XML schemas for vertical industry standards are full-featured indeed.

Finally, I would be interested in hearing about any research into whether the overall throughput of a web application is increased or decreased by the introduction of JSON down to the client. It's nice that parsing gets a hundred times faster on the client, but when you're server is trying to handle more than a hundred concurrent users, and the server has to translate from XML to JSON (and back again), then it sure seems like the use of JSON is pushing processing burden in the wrong direction.

Data-driven Table User Interfaces

Submitted by administrator on Sat, 04/24/2010 - 00:27.

A customer asked me recently how they could use XForms constructs to create a form that dynamically populates a table with available products that can be ordered based on selection of a product provider. In this case, the customer also wanted to have the product order list be editable once provided, allowing the user to delete rows or even to add more rows to the product order table initially provided for the selected store. Seemed like another good example for the blog.

So, suppose you have a main data instance for a form that looks something like this:

<xforms:instance id="data" xmlns="">
<data>
     <storeID></storeID>
     <products>
        <store>
           <product name="" code="" cost="0" qty="0" total="0"/>
        </store>
     </products>
     <total>0</total>
</data>
</xforms:instance>

Each product has a name, code, and cost, and the end-user will indicate the quantity of the product they desire. The above data corresponds to showing an empty table until the user chooses a "storeID". Here is the XForms user interface markup for showing a four column table having as many rows as there are "product" elements in the data.

<xforms:repeat nodeset="products/store/product" id="orderTable">
   <xforms:group ref=".">
      <xforms:input ref="@name">
          <xforms:label>Product Name</xforms:label>
      </xforms:input>
      <xforms:input ref="@cost">
          <xforms:label>Unit Cost</xforms:label>
      </xforms:input>
      <xforms:input ref="@qty">
          <xforms:label>Desired Quantity of Product</xforms:label>
      </xforms:input>
      <xforms:input ref="@total">
         <xforms:label>Calculated Line Total</xforms:label>
      </xforms:input>
   </xforms:group>
</xforms:repeat>

Initially, the table will have just one row of four user interface elements containing essentially empty or zero values. However, now lets hook up something that allows us to pick a store ID so we can fill the table with an initial order of products. Now, it would be reasonable in a full form application to obtain the list of stores from a web service and then get the starting list of products for the selected store from another web service. Getting data from web services is not an important but orthogonal point, so in this mock-up, I'm going shorten all of that down to just having the data available in a format that looks like this:

<xforms:instance id='storeLists' xmlns="">
   <stores>
      <store ID="A">
        <product name="Widget" code="W1" cost="3.50" qty="0" total="0"/>
        <product name="Gadget" code="G1" cost="4.25" qty="0" total="0"/>
        <product name="Trinket" code="T1" cost="2.75" qty="0" total="0"/>
      </store>
      <store ID="B">
        <product name="Gadget" code="G1" cost="4.25" qty="0" total="0"/>
        <product name="Gromet" code="G2" cost="3.50" qty="0" total="0"/>
      </store>
      <store ID="C">
        <product name="Widget" code="W1" cost="3.50" qty="0" total="0"/>
        <product name="Trinket" code="T1" cost="2.75" qty="0" total="0"/>
        <product name="Gromet" code="G2" cost="3.50" qty="0" total="0"/>
        <product name="Sprocket" code="S1" cost="1.99" qty="0" total="0"/>
      </store>
      <store ID="D">
        <product name="Locket" code="L1" cost="9.50" qty="0" total="0"/>
        <product name="Pocket" code="P1" cost="1.50" qty="0" total="0"/>
        <product name="Rocket" code="R1" cost="7.50" qty="0" total="0"/>
        <product name="Sprocket" code="S1" cost="1.99" qty="0" total="0"/>
        <product name="Socket" code="S2" cost="2.49" qty="0" total="0"/>
      </store>
   </stores>
</xforms:instance>

The user is provide the ability to select a store using the "select1" control, and the list of stores can be easily picked up from the data using an "itemset". Once the user makes a choice, an "xforms-value-changed" event on the select1 could be used to run a web service to get the product list, but here we'll just mock that up with an "insert" because the data is already available:

<xforms:select1 ref="storeID" appearance="minimal">
<xforms:label>Choose a store</xforms:label>
<xforms:itemset nodeset="instance('storeLists')/store">
     <xforms:label ref="@ID"/>
     <xforms:value ref="@ID"/>
</xforms:itemset>
<xforms:action ev:event="xforms-value-changed">
     <xforms:delete nodeset="instance('data')/products/store" at="1"/>
     <xforms:insert context="instance('data')/products" origin="instance('storeLists')/store[@ID = instance('data')/storeID]"/>
</xforms:action>
</xforms:select1>

The "ref" on the select1 tells where to store the resulting store selection. The "nodeset" on the itemset tells where to get the list of stores from. The "ref" attribute on the xforms:label in the itemset tells what to show for each item in the list of choices, and the "ref" xforms:value tells what to store in the data ("storeID" due to the ref on select1) when a particular list choice is selected.

The "xforms-value-changed" event handler recognizes when a selection has been made, since that results in a value change on the "storeID" data node. The delete action gets rid of any preceding list, and then the insert action copies the list of products for the selected store into the main data. In particular notice that the XPath predicate in the origin attribute selects a store element to copy based on the store element's ID attribute matching the selected store identity placed in the storeID element by the value change behavior of the select1.

Once this insert occurs, the xforms:repeat is automatically responsive to the change of the data. It generates a four column row of user interface controls for each of the inserted product elements. For example, if the user picks store A, then they get three rows for Widgets, Gadgets and Trinkets. If they then pick store D, the form automatically adjusts to five rows for Lockets, Pockets, Rockets, Sprockets and Sockets.

Once the table content has been set with the product list for a particular store, the user may choose to add or delete rows from the table. Here is an additional instance that would be used to store the data prototype for a product:

<xforms:instance id='proto' xmlns="">
<prototypes>
<product name="" code="" cost="0" qty="0" total="0"/>
</prototypes>
</xforms:instance>

A button to add a row to the repeat table would trigger the addition using the following XForms markup:

<xforms:trigger>
<xforms:label>Add Item</xforms:label>
<xforms:action ev:event="DOMActivate">
     <xforms:insert context="products/store" nodeset="product"
                    at="index('orderTable')" position="after"
                    origin="instance('proto')/product"/>
     <xforms:setfocus control="orderTable"/>
</xforms:action>
</xforms:trigger>

This simply inserts a product prototype, obtained via the origin attribute, into the location defined by the context and nodeset attributes at a position corresponding to the row of the table that current has the input focus. When the data is inserted, the repeat table automatically generates another four-column row of user interface elements to allow the user to interact with the new data.

Similarly, a button to delete a row from the repeat table would trigger the deletion using the following XForms markup:

<xforms:trigger>
<xforms:label>Delete Item</xforms:label>
<xforms:action ev:event="DOMActivate">
     <xforms:delete context="products/store" nodeset="product" at="index('orderTable')"/>
     <xforms:insert context="products/store" at="1" position="before" origin="instance('proto')/product"
                    if="count(instance('data')/products/store/product)=0"/>
     <xforms:setfocus control="orderTable"/>
</xforms:action>
</xforms:trigger>

This simply deletes the product data element corresponding to the row of the table that currently contains the input focus. The row of user interface elements that presented this data is automatically deleted. As the next step of the action script, if the product list data becomes empty due to the prior delete, a new empty product prototype is inserted. The repeat table then presents one row of interface elements, so this extra insert ensures the user is never left with an unsightly empty table.

Last but not least, the actions scripts of both of the triggers above end with an xforms:setfocus action. This is because pressing a button, be it to add or delete an item from a table, transfers focus to the button. That's just how the web works. But the user's focus is not really on the buttons; those are just tools. The user's focus is on changing the table, so it is a better user experience to push the focus back to the repeat table.

Creating a Dynamic Questionnaire Template with a Lotus Form

Submitted by administrator on Wed, 03/24/2010 - 23:49.

As an example of the powerful data-driven dynamism available in Lotus Forms due to features of XForms, I'd like to take you through a brief conceptual tour on the focused example of creating a Lotus Form template for a Questionnaire or Survey. This template is able to handle not just any number of questions and any amount of question text, but also any kind of answer type. And all of this would be controlled by the data so that the actual design of the Lotus Form template is the same.

The power of being purely data-driven should not be glossed over. You can easily have web application servlet code that obtains the questionnaire template and then prepopulates it with specific questionnaire data so that the client side receives a specific questionnaire selected in a previous step of the web application. But, XForms-based Lotus Forms also have that AJAX property of being responsive during run-time to new data obtained by a form via web services or other http submissions. So, you could even have a Lotus Form that obtains and adds new questions on the fly in response to answers provided to initial questions.

This post will focus on the main repeating template that provides the dynamic presentation layer for each question of a questionnaire or survey. As this is an example of a purely data-driven questionnaire, let's start by looking at a sample data format. Suppose you have a survey consisting of any number of items, each of which can contain a question text, an indication of the type of question being asked, a place for an answer, and optionally some possible choices for those answers. Something like this:

<survey>
<item>
     <question type="yesno">Do you like apples?</question>
     <answer></answer>
</item>
<item>
     <question type="likertscale">It is OK for apples to have a powdery texture.</question>
     <answer></answer>
</item>
<item>
     <question type="closedselection">What is your physical gender?</question>
     <answer></answer>
     <choices>
        <choice label="Female" code="F"/>
        <choice label="Male" code="M"/>
     </choices>
</item>
...
</survey>

In the XFDL presentational language that Lotus Forms combines with the XForms data processing layer, every XForms user interface element has a container XFDL element
for presentation. The survey format consists of a number of <item> elements, so an XFDL <table> containing an <xforms:repeat> is the correct top-level presentation element:

The table has a scope identifier (sid) attribute that allows the table to be programmatically referenced, but we won't be using that feature in this example. The table can also have XFDL options outside of the <xforms:repeat> to control presentational aspects like borders and background colors, and we aren't focusing on that either.

The <xforms:repeat> has an attribute called "nodeset" which uses an XPath expression to make a reference to however many <item> elements are in the <survey>. This is an automatic or "declarative" loop construct. For each <item> node in the data, no matter how many there are, the template content of the <xforms:repeat> is generated to present that <item> to the user. Even if new <item> elements are added at run-time, e.g. by a web service or an <xforms:insert> action, the XFDL table in the Lotus Form will dynamically grow to present the new <item> elements. And even if some <item> elements are removed from the data, e.g. by an <xforms:delete> action associated with an XFDL <button> by an <xforms:trigger>, the XFDL table will dynamically and automatically remove the corresponding user interface elements that were presenting those removed <item> elements.

So, the magic really happens in the template inside the <xforms:repeat>. In Lotus Forms, you can put any and all kinds of XFDL items in the <xforms:repeat>, including more XFDL table items. In this example, we will be showing a few variations that present different kinds of user interface controls for collecting a few different kinds of answers to questions.

First off, though, presenting the actual question text for an <item> is a simple matter of using an XFDL label item with an <xforms:output>, like this:

<xforms:repeat nodeset="/survey/item">
   <label sid="QuestionText">
      <xforms:output ref="question"/>
      ... 
   </label>

   <layoutflow>block</layoutflow>

   ... 
</xforms:repeat>

For each survey <item>, an XFDL <label> item is generated, and it binds to the <question> child element of that associated <item> using the "ref" attribute. The XFDL label item presents the text of the bound <question> node, and other XFDL options can be used to provide styling such as the block layout flow as well as alternative font color, background color, font selection and so forth.

More XFDL items can be added to the <xforms:repeat> to collect the answer for the given question. In many cases of XFDL tables, each XFDL item within the <xforms:repeat> template is actually presented to the user. An example would be using each XFDL item in the <xforms:repeat> to represent one column of a purchase order table. However, it is not necessary to show all of the XFDL items within the <xforms:repeat> template. In fact, XForms user interface controls have a selective binding feature that XFDL items support, since the XFDL items are wrappers for the XForms user interface elements.

The selective binding feature of XForms will be used to help easily choose one XFDL item from among many to collect the user's answer to the question. Each question can have a different type of answer, so each "row" of the table can make a different choice of user interface control used to collect the answer. The selective binding feature uses an XPath predicate to decide whether or not the XForms user interface element binds to a node of data or not, and the control is invisible if it is not bound to a data node.

In the example survey data above, the first <item> contains a <question> whose type attribute indicates it is a "yes/no" question. Inside the <xforms:repeat> we can create a checkbox item that can collect a (schema valid boolean) true/false answer, as follows:

The above checkbox widget only binds to <answer>, and therefore is only visible, if the corresponding question type is 'yesno'. Otherwise, the XPath in the ref attribute of the <xforms:input> does not select any nodes, so the XFDL <check> item is not visible.

The second <item> of sample data above has a type of 'likertscale', so we would like to show a 5-point radio button group rather than a checkbox. As explained above, the check box on the second row of the survey table automatically hides itself due to selective binding, so all we have to do is add an XFDL <radiogroup> item to the <xforms:repeat> to provide the interface for collecting the 'likertscale' type of answer, as follows:

<radiogroup sid="Answer_likertscale">
   <xforms:select1 ref="answer[../question/@type='likertscale']" appearance="full">
      <xforms:label/>
      <xforms:item>
         <xforms:label>Strongly agree</xforms:label>
         <xforms:value>1</xforms:value>
      </xforms:item>
      <xforms:item>
         <xforms:label>Agree</xforms:label>
         <xforms:value>2</xforms:value>
      </xforms:item>
      <xforms:item>
         <xforms:label>Neither agree nor disagree</xforms:label>
         <xforms:value>3</xforms:value>
      </xforms:item>
      <xforms:item>
         <xforms:label>Disagree</xforms:label>
         <xforms:value>4</xforms:value>
      </xforms:item>
      <xforms:item>
         <xforms:label>Strongly disagree</xforms:label>
         <xforms:value>5</xforms:value>
      </xforms:item>
   </xforms:select1>
   ...
</radiogroup>

The third survey <item> in the sample data above provides a closed selection of choices. That could be styled using a pair of radio buttons, a pair of mutually exclusive checkboxes, a list box, or a popup control that provides a simple dropdown list. The answer types in the survey format could be made to distinguish these possibilities using more keywords, but for this example we'll just assume that a <popup> control is the desired presentation for a closed selection. The XFDL markup below shows how this can be done, and it is also interesting because it is shows that the data can also dynamically control the choices, rather than having only static choices as shown in the <radiogroup> above.

It seems a useful, now, to round out this blog post by presenting a few more examples for other common types of input, such as single-line strings, multiline text, and dates. Here's what the data would look like:

<survey>
...
<item>
     <question type="oneline">What's your name?</question>
     <answer></answer>
</item>
<item>
     <question type="multiline">Do you have any other comments?</question>
     <answer></answer>
</item>
<item>
     <question type="date">What is your date of birth?</question>
     <answer></answer>
</item>
...
</survey>

The corresponding XFDL items that would be added to the <xforms:repeat> content template for these types of questions would be:

So, hopefully you now have the idea that a completely dynamic and completely data-drive survey or questionnaire can be created using the features of XForms in XFDL (Lotus Forms). Any number of XFDL items can be added to the <xforms:repeat>, XPath predicate selection can be used to choose one XFDL item from among many to collect an answer for a survey question, and most importantly that a different choice of user interface control can dynamically selected for each survey question.

Spotlight on XForms 1.1 Features in Lotus Forms

Submitted by administrator on Fri, 03/05/2010 - 02:12.

Lotus Forms has supported XForms for a number of years now, and you can get a good idea of all the features supported from the XFDL reference manual.

However, now that XForms 1.1 has been finalized, I've had a number of questions about shining a spotlight on the XForms 1.1 specific features in Lotus Forms. Quite a number of XForms 1.1 features were improvements to the semantics of pre-existing XForms 1.0 features, and no small number of those improvements were based on feedback from the IBM Victoria Software Lab, so obviously we implement those and it would be too long to go into them. The spotlight will be on Lotus Forms features syntactically activated with new XForms 1.1 vocabulary that was not available in XForms 1.0.

One of the coolest and most powerful additions to XForms 1.1 were the if and while attributes on XForms actions. XForms actions are behaviors like changing data values, insert or deleting nodes, or making web service calls, and they can be set to happen in response to events like a button press (DOMActivate) or user input (xforms-value-changed). The if and while attributes enable XForms actions to be conditionally or iteratively executed when these events occur.

Lotus Forms supports the context and origin attributes on insert and delete actions. These attributes enable handling of empty repeating data, and they allow repeating data to be copied from a data template. It is much easier to handle dynamic table data with these features. Also, when deleting a data node representing a row of a table, if the table becomes empty, then the above if attribute can be used on an insert to detect that this has happened, and insert a new empty data node. The net result observed by the user is that deleting the last row of a table looks like it just clears out that row so that the user can start immediately entering more data.

Lotus Forms also implements the XPath function compare(), which means a form author could use XForms actions with the if and while attributes to sort or search data, if the need arose. Several other functions are implemented, including:

random() - in case you want to write a Lotus Form that plays Black Jack
current() - to help with data table lookups
power() - for exponential calculations such as compounded interest payments
days-to-date() - can be used in combination with days-from-date() to do simple date math like "today plus 90 days"
seconds-to-dateTime() - can be used in combination with seconds-from-dateTime() to do dateTime math like "now plus 3 hours"
local-date() - provides the date for the end-user, rather than the UTC date
local-dateTime() - provides the end-user date and time, rather then the UTC date and time.

Lotus Forms supports the display of images obtained from XForms instance data, in both button and label items, using the mediatype="image/*" attribute setting on xforms:output.

Lotus Forms supports the xforms:* datatypes, which allow an empty string to be valid on the corresponding xsd:* datatypes, like xsd:date. Whereas pure XML schema datatype definitions are intended to define what constitutes valid completed data, this feature of XForms recognizes the importance of a good user experience before and during completion of the form.

Finally, Lotus Forms supports several of the new features of xforms:submission, including:

the method="put" and method="delete" attribute settings to round out access to ATOM publishing services
the relevant and validate attributes, which allows a submission to turn off data validation and relevance pruning. This can be used to implement a "Save to Server" capability so that a user can perform a fill experience over multiple sessions.
the serialization="none" attribute setting to enable an xforms:submission to perform simple URL activation.
The targetref attribute, which enables a web service call to replace only a portion or subtree of a data instance.
The replace="text" attribute setting, which allows a web service call to replace the content of the target data node, rather than the data node itself. This is useful for accessing web services that return textual content rather than XML.

Lotus Forms also has a number of XFDL extensions that add value to the integration with XForms, but that is the subject of another blog for another time.

XForms: binding to an optional element

Submitted by administrator on Wed, 01/20/2010 - 08:06.

I asked this on the XSLTForms list, but it’s worth casting a wider net.
Say I have an XML instance like this:
<root>
<format>xml</format>
</root>
Possible values for the format element are “xml”, “text”, “binary”, or a default setting, indicated by a complete absence of the <format> element. I’d like this to attach to a select1 with four choices:
<xf:select1 ref=”format”>…
* [...]

Treating URIs as strings considered dangerous

Submitted by administrator on Mon, 01/18/2010 - 12:41.

Since URIs are often conveyed as strings it's tempting to manipulate them as such, but it's better--and safer--to delegate URI manipulation to special functions. These can then have their own unit-tests, which will take into account the edge-cases that can catch us out.

I've just been doing a quick code review in the Ubiquity XForms project, and one thing caught my eye that I thought might be worth a post.
Forms submission
In forms submission -- both in XForms and HTML forms -- we often need to add parameters to a URI.
For example, if we have the URI http://example.org, and the parameters a=b and c=d, then the resulting URI should be:

http://example.org?a=b&c=d

It seems pretty straightforward that we need to add the parameters to the URI, with a '?' in between:

[uri] + '?' + [parameters]

Adding parameters
We can see that the parameters themselves have been added by taking the name and value (a=b, etc.), and adding it to the URI, ensuring that for all parameters other than the first, there is a separator between. (The separator can be either an '&' or a ';'.)
Base URIs already containing parameters
The bug that needed fixing though, was that this 'naive' concatenation doesn't work if the URI you are dealing with already contains parameters.
For example, if we have the URI http://example.org?x=y, and we need to add the same two parameters we had before, then our simple concatenation would give us:

http://example.org?x=y?a=b&c=d

when what we actually need is:

http://example.org?x=y&a=b&c=d

As you can see, if we already have a '?' then we don't need to add another, so it seems that a simple addition to our concatenation code would be to use a call to indexOf to see if there is a '?' present, and only add another if we don't find one.
There's a small additional test we'll need to make which is to check whether the last character in the URI is a '?', as I'll explain.
Base URIs with empty query strings
Recall that we added the parameters by placing a=b and c=d onto the end of the URI, separated by '&':

http://example.org?a=b&c=d

Now, if we already have a URI with a query then we need to ensure that there is an additional '&' placed before our first parameter:

http://example.org?x=y&a=b&c=d

But what if the base URI has a query indicator (i.e., the '?') but no parameters? In other words, what if we have this URI:

http://example.org?

In this situation we don't want to add the extra separator, otherwise we'll get this:

http://example.org?&a=b&c=d

So our rules now become that we only want to precede our parameters with a separator if the '?' is not the last character in the URI. It's a little awkward, but thanks to lastIndexOf, I'm sure we can manage.
Base URIs already containing fragments
However, there's a further subtlety; what if the URI contains a fragment identifier?
For example, if we have the URI http://example.org#x, and we need to add the same two parameters we had before, then our simple concatenation would give us this:

http://example.org#x?a=b&c=d

The fragment identifier part of the URI has now become x?a=b&c=d because it's always the last part of the URI. What we actually want is to insert the new parameters before the '#':

http://example.org?a=b&c=d#x

Now we need to add another use of indexOf to check for a '#', and if we find one, use its position as the point at which to insert the parameters.
Context is everything
However, the assumption behind using indexOf and lastIndexOf in this way is that a URI will contain only one '?'. A secondary assumption here is that the only time you'll ever see a '? is as an indicator of the query part of the URI.
Both of these assumptions are incorrect.
Question marks in parameters
The first assumption is that you can only have one '?' in a URI. However, the query section of RFC 3986 explicitly flags up that the '?' character is a valid parameter value. For example, we can have a=finished? as a parameter.
This means it's quite easy to envisage scenarios where there is more than one '?' in the URI:

http://example.org?a=finished?&c=d

This won't necessarily mess up our first use of indexOf, but it will mess up the use of lastIndexOf as a way to check whether you need to add an extra separator. Recall that we wanted to avoid turning this:

http://example.org?

into this:

http://example.org?&a=b&c=d

so we used lastIndexOf to check whether the last character in the URI was a '?'. But that algorithm will turn this:

http://example.org?a=finished?

into this:

http://example.org?a=finished?c=d

You might need to look closely to spot it, but because the last character was a '?', we haven't added a separator before the c, and as a result, instead of having two parameters (a=finished? and c=d) we have only one (a=finished?c=d).
Question marks in fragments
The interesting thing about the previous examples is that at least you know you have a query string, so you might be tempted to still use indexOf to manipulate things. After all, although we may have too many '?' characters, we still know that we have query.
However, with the the fragment section of RFC 3986 all bets are off; here we can see that '?' is explicitly allowed as a fragment character.
This means that it's possible to have a '?' in a URI even if it doesn't have a query. For example:

http://example.org#finished?

This may seem like a contrived example, but actually it's not, for two reasons.
The first is that the fragment part of a URI is carefully defined to allow anything, because we don't know how it will be interpreted. You may think that "finished?" is not valid as an HTTP fragment, but what about in the scheme "xyz"?
And this is the key point; since HTML forms and XForms can ultimately deal with any scheme, since that's how the web is designed, we must write our algorithms defensively, and not assume anything.
Safe URI handling
Hopefully this delving into some of the subtleties of URI handling and parsing--and we haven't even begun to talk about turning relative paths into absolute paths, handling encoded characters, and so on-- has done enough to convince you that you shouldn't manipulate URIs directly, as simple strings.
The only way to be completely sure of what is happening is to use special functions to unpack a URI into its various components, then manipulate those components--perhaps adding additional parameters to the list of query parameters, but it might also be to turn a relative path into an absolute path--before finally reassembling the URI.
This may sound like a lot of work, but it's the only way to be sure that characters don't get incorrectly interpreted as a consequence of their position in the URI not being taken into account.
In the backplanejs library these functions are in the URI module. (This is also imported into the Ubiquity XForms library.) Breaking up a URI simply involves calling spliturl, which returns an object containing all of the parts. For example:

spliturl( "http://example.org?a=finished?" )

would give us the object:

{
scheme: "http:,
authority: "example.org",
path: ""
query: "a=finished?"
fragment: ""
}

It's then an easy matter to manipulate the query part, before creating a new URI with the recomposeURI method.
Conclusion
Since URIs are often conveyed as strings then it's tempting to manipulate them as such. But the problem with doing this means that the context of a character is rarely taken into account when processing.
It's better--and safer--to delegate URI manipulation to special functions. These can then have their own unit-tests, which will take into account the edge-cases that can catch us out.

Looking forward to XForms 1.2 Progress

Submitted by administrator on Mon, 01/04/2010 - 19:47.

As we start off the business of a new decade, I already find myself looking forward to the improvements in XForms 1.2.

I know, I know, we just released the XForms 1.1 Recommendation, which contained a huge bundle of new features and architectural improvements. Still, the working group has been energetically advancing on still more features and improvements even as we closed the loop on the W3C process in the latter half of last year. I am very pleased with the completion of XForms 1.1, but the only constant is change, and we have an excellent array of new technical results to achieve in XForms.

The working group has put a fair bit of effort into the triage of features to separate nearer term (and simpler) objectives from those features of a more architecturally significant nature, which tend to get put in the "XForms 2.0" bucket.

My personal favorite "simple" feature is model-based UI switching with the switch element. This may seem like a bit of a nerdy thing to perseverate on, but the switch element is the only UI control we have that is not directly controllable by a UI binding to data. So, if a form author wants to switch UIs based on a user input, such as changing the payment details UI based on the selection of a payment, then the form author typically resorts to using group elements and the relevant model item property. But this is more than just the sense of completion we'll get from being able to say *all* UI controls can be declaratively linked to data. And it's more than just the satisfaction we'll get from providing the right tool for the job. The ability to declaratively link the state of a switch to data is critical to the save/reload web application pattern and the transaction digital signature web application pattern.

Other web application standardization efforts are focused mainly on what happens during execution of a single web page. XForms-based technogies are a step ahead because they can more easily address the user patterns related to collaboration on the web. Whether it is one user saving an application context and reloading it again in the future, or whether one user provides data to be loaded by a second user, the simple fact remains that data can be in any state along a continuum between empty template and fully completed, so web applications need to be declaratively responsive to that continuum. And nowhere is this clearer than when a web-based transaction is of high enough value to warrant a digital signature. A digital signature needs to be able to capture the full current state of the web application, and having a declarative binding to the data is the easiest way to achieve that goal since it means one can simultaneously sign the web application page template(s) and the data as separate resources.

For the record, Lotus Forms has for a few years now used a legitimate XForms extension mechanism, the xfdl:state attribute, to support model-based switching. We deemed this extension critical to our ability to address our customers' requirements for the above two web application patterns. Standardized integration of XForms with XML Signatures and other advanced security features like XAdES is, so far, in the XForms 2.0 bucket.

In conclusion, I'm not necessarily saying this is the "biggest" feature of XForms 1.2, only that it is my fave. I'll be giving some coverage to the other XForms 1.2 features in future blog posts, so please click the 'follower' button above (top right) and stay tuned.

The Model Endpoint Template (MET) organizational pattern for XRX apps

Submitted by administrator on Sun, 11/29/2009 - 21:09.

One of the lead bullets describing why XForms is cool always mentions that it is based on a Model View Controller framework. When building a full XRX app, though, MVC might not be the best choice to organize things overall. Why not?
Consider a typical XRX app, like MarkLogic Application Builder. (You can download a your [...]

Application Builder behind-the-scenes

Submitted by administrator on Wed, 10/21/2009 - 19:04.

I’ll be speaking next Tuesday (Oct 27) at the Northern Virginia MarkLogic User Group (NOVAMUG). Here’s what I’ll be talking about.
Application Builder consists of two main parts: Search API to enable Google-style search string processing, and the actual UI wizard that steps users through building a complete search app. It uses a number of technologies [...]

XForms 1.1 is out

Submitted by administrator on Wed, 10/21/2009 - 07:01.

XForms 1.1 is now a full W3C Recommendation. Compared to version 1.0, which went live a bit more than 6 years ago, version 1.1 offers lots of road-tested tools that make development easier and more powerful, including new datatypes and XPath functions, a significantly more powerful submission subsystem, and a more flexible event model.
And XSLTForms [...]

XForms 1.1 is a W3C Standard

Submitted by administrator on Tue, 10/20/2009 - 21:24.

The theme song to Star Trek started to play in my head last night when I thought about the imminent publication of XForms 1.1 as a W3C standard (which the W3C calls a Recommendation). It's been five years since I started putting the working group's content into the first "thin spec" working draft of XForms 1.1.

Since that time, the working group has substantially upgraded XForms to enable it to better address the emerging requirements of cutting edge web applications and interactive documents. This has included well over a thousand spec changes to resolve differences in understanding both great and small. Many times it has happened that one person's editorial clarification is another's substantive change, so there was a significant improvement in mindshare and consensus at a much deeper level than was the case for XForms 1.0.

The biggest change, in my view, is the substantial increase in web connectivity features added to the XForms submission module. This enables XForms to properly act as a smart Web 2.0 application client. XForms handles the interactive rich user experience of the web application, and at any point in that interaction, XForms can directly inject information into and extract information from server-side data sources and business processes exposed as web services, REST services, ATOM feeds and publishing services, etc.

As powerful as these features are when used within a web application's HTML pages, they are all the more interesting when used to connect open standard ODF office-style documents to data sources and business processes. Then, you get the blend of ease-of-user and familiarity features of a word processing or spreadsheet document and the desirable connectivity features of a Web 2.0 application. For further details on these thoughts, please see my document engineering conference papers on ODF and XForms: "Interactive Office Documents: A New Face for Web 2.0 Applications", "An Office Document Mashup for Document-Centric Business Processes", and "Enriching the Interactive User Experience of Open Document Format" (these papers are likely to be available to you through your institutional account with the ACM, e.g. through KnowledgeGate for IBMers).

There are many other additions and refinements in XForms 1.1, which are summarized in the introduction. Suffice it to say that five years of effort have gone into making XForms ready for prime-time, and we at IBM are very pleased to see this version of XForms advance to W3C Recommendation, especially due to the number of products we already have in market that support this computing industry standard. With XForms 1.1, we will soon be able to realize the vision of delivering Web 2.0 applications via "A REST Protocol and Composite Format for Interactive Web Documents".

Understanding the XForms dependency engine

Submitted by administrator on Wed, 09/23/2009 - 12:04.

A key component of any XForms processor is the dependency-engine. The idea is pretty straightforward, and will be familiar to anyone who has used a spreadsheet; if some item has its value set by a calculated expression that contains references to other items, then when any of those items change, the first item must be recalculated.

Read more in my introductory article on the XForms Developer Zone site, Understanding the XForms dependency-engine.

Countdown to Safari

Submitted by administrator on Wed, 09/23/2009 - 10:01.

The next big task that I'm about to tackle on the Ubiquity XForms project is support for Safari and Chrome. In general, UXF has seemed to work pretty well in the WebKit-based browsers whenever I've tried it, but I've never run exhaustive tests.

So yesterday, for the first time, I ran the full XForms 1.1 test suite in Safari. The results were a huge disappointment: 443 tests were executed, of which 402 failed and just 41 passed. Suddenly, the job ahead seemed colossal.
After my initial shock wore off, and feeling dubious about the figures, I decided to watch the tests more closely as they were being run in Selenium. It quickly became apparent that a problem existed in our isModelReady() extension function, because Selenium was frequently executing its commands before the test forms had finished loading. Sure enough, some poking around revealed a bug that was easily resolved.
I'm pleased to be able to say that, post-commit, the test suite results for Safari are a lot more promising. Of the 443 tests that ran, 195 failed and 248 passed; a 55% (ish) pass rate. Not great, but far less daunting as a starting point. Among the failures, there look to be a few clumps of tests that may be of common origin, which bodes well for the possibility of quick wins. Submission looks like being the biggest single area that needs work, as chapter 11 failed almost in its entirety.
Issues have been raised for many of the failures in the bug tracker. So if anyone else feels like getting their hands dirty, feel free to take some. The long road to Safari support starts here!

XForms Developer Zone

Submitted by administrator on Wed, 09/23/2009 - 06:14.

Another XForms site launched this week. This one seems pretty close to what I would like XForms Institute to become, if I had an extra 10 hours per week. -m

Interactive Web Documents use XForms and REST to Create Sophisticated Web 2.0 Applications

Submitted by administrator on Fri, 09/18/2009 - 08:13.

Yesterday I presented a full paper on a new method and research prototype that describes our thinking on the strategic technical direction of the Lotus Forms document format and processing model. The paper is now available from the ACM Digital Library and is entitled "A REST Protocol and Composite Format for Interactive Web Documents". If you have one, use your institutional access to the ACM Digital Library. For IBMers, you can get access using KnowledgeGate.

As you may know, the document format for Lotus Forms is called XFDL, and it has grown and changed many times in the last 15 years to become more and more comformant to open standards. Once upon a time, back in the 1990's, XFDL was not even an XML format. Currently, the XFDL format is an XML vocabulary that describes a high precision multipage layout for an interactive Web 2.0 application, and XFDL incorporates other standards including XForms, XML Schema and XML Signatures to express dynamic behaviors, business rules, client-side data validation, web service access, and digital signature security.

This open standards transition has represented an important and substantial evolution of the Lotus Forms document format and processing model over time, and this new paper on "Interactive Web Documents" articulates the next step in the evolution of Lotus Forms. As with any strategic operation, it is the kind of thing that plays out over years in the product line, but I am used to that, and I am very excited by our progress.

Under this "interactive web document" transition, the multipage capabilities of Lotus Forms would be offered using multiple HTML or XHTML web pages aggregated using the ODF packaging format. The XForms capability would be offered on the client-side using the Ubiquity XForms processor. Many of the fine-grain features that add significant business value to XFDL would be implemented using javascript or small stateless web service calls as appropriate. For example, the full power of the ICU for access to locale-specific information would be accessed via a web service. As well, the XML signature functionality would be offered almost entirely via a server-side signing engine (except for the final client-side private key operation to generate a value for the dsig:SignatureValue element).

The initially empty interactive web document template would describe the web application. To initiate a business process, the template is instantiated, becoming an interactive web document capable of holding XML data and file attachments provided by end-user participants and machine agents in the business process. The instantiation sets up a REST service keyed by a unique ID to the instantiated document, and simple GETs, PUTs and DELETEs are used to access and update the resources in the composite document. In particular, XForms submissions are used to synchronize data and change pages within the interactive web document. When a user completes their work on the document, it is the edit token, not the whole document that is passed along through the business process. In this way, the interactive web document and its REST interaction session become the central artifacts over which the entire business process operates, with document serialization only being required to transmit the interactive web document (i.e. the web application combined with its current state) across enterprise domain boundaries.

This is the general idea, and the paper contains numerous additional technical explanations and markup examples. It should be clear from this description, though, that this next evolutionary step will enable Lotus Forms to more easily adapt to and incorporate emerging Web 2.0 technologies, via its page basis in HTML and XHTML, and it will provide a quantum leap in performance of the Webform Server by avoiding a single large XML DOM for large forms applications, by offloading XForms processing to the client-side, and by avoiding serialize/parse operations during the course of the business process.

XForms Developer Zone and User Group launched

Submitted by administrator on Wed, 09/16/2009 - 12:00.

We're pleased to be launching two new initiatives to help people who are interested in XForms.

The first is the all new XForms Developer Zone web-site -- or xformsdz, as we're calling it.

Whilst the Developer Zone will be unashamedly biased towards XForms, within that, we'll have discussions, articles, code snippets, and tutorials about any XForms processor we can find, and any application framework in which it's used.

To accompany the web-site, we're also launching a regular newsletter, and a London XForms User Group.

Read more at XForms Developer Zone and User Group launched on my webBackplane blog.

XForms Developer Zone and User Group launched

Submitted by administrator on Wed, 09/16/2009 - 04:00.

We're pleased to be launching two new initiatives to help people who are interested in XForms.
The first is the all new XForms Developer Zone web-site -- or xformsdz, as we're calling it.
Whilst the Developer Zone will be unashamedly biased towards XForms, within that, we'll have discussions, articles, code snippets, and tutorials about any XForms processor we can find, and any application framework in which it's used.
To accompany the web-site, we're also launching a regular newsletter, and a London XForms User Group.
The London XForms User Group
The inaugural meeting of the user group will be in Clerkenwell, London, hosted by Skills Matter. We'll be looking at the architecture of XForms, in particular what it means to have MVC support built in from the ground up, and we'll be using XForms and Google Maps as a case-study.
The October meeting will look at the benefits of using XForms for the UK insurance industry, and we're lucky enough to have an industry expert -- Neal Champion -- speaking to us that evening.
If you'd like to come along, we'd love to see you. And even if you can't make it, join the user group mailing-list, and we'll keep you posted about XForms-related activities and events, in and around London.

Announcing New Release of Ubiquity XForms

Submitted by administrator on Tue, 09/01/2009 - 23:20.

Ubiquity XForms version 0.7 is now available. In addition to contributing to the recent advancement of XForms 1.1 to Proposed Recommendation, this new version has many new features, fixes and highlights. Personally I'm happiest with the progress on implementing submissions, but that may be a bit of bias because I contributed some code to deal with aspects of submissions such as serialization, validation and relevance pruning. On the other hand, I contributed code in other areas, so maybe it's just that I'm biased toward any improvements to our ability to support the XRX architecture and connect the XForms client to server-side services.

In any case, there's lots of good core XForms features that have been added as well as noteworthy extension features. My personal favorites are the calendar and checkbox controls used for date and boolean type XForms inputs, the new map controls, and the very simple javascript function extension. Check out the loan-lowercase.html sample and its format.js file, for example.

And this blog entry would not be complete without my mentioning a special word of appreciation to the folks at webBackplane for all their contributions in general but especially for the Ubiquity XForms rollup system, which consolidates the many files of the processor into a single file to be deployed on your server. I have an internal project right now that uses the rollup, and it provides us with the high level of performance we expected/required in our applications.

Full details of the version 0.7 release can be found here: <http://code.google.com/p/ubiquity-xforms/wiki/Release0Point7>

Linking issues and revisions in Google Code

Submitted by administrator on Sun, 08/23/2009 - 13:18.

We use Google Code a lot in our projects, because it provides a great range of tools, with a straightforward interface.
It's particularly useful to have version control and issue-tracking in the same system, and it's very easy to cross-reference the two by referring to revisions in issue comments, and to issues in the commit messages.
On top of that, Google Code provides some nice code review facilities, and here too, it's possible to refer back to revisions and issues.
In this post I'll outline how we use these tools together in UXF, the Ubiquity XForms AJAX library.
Everything should have an issue
The first thing that's needed is an issue. This might be an issue in the traditional sense, of a bug. But it might also be a desirable feature, that someone would like to add. Whatever it is, we shouldn't really be starting to code until we've written down what we're going to do, or what problem we're trying to solve.
Let's use issue 515 as an illustration. At the top we see it has the following description:

We need a UI control that can adhere to data binding restrictions for
xsd:boolean / xforms:boolean. The control may render a checkbox, for
example.

Sample code snippet:

<xf:input ref="check" datatype="xforms:boolean">
<xf:label>Check: </xf:label>
</xf:input>

Based on conversation with John in last week's call, needed in this
iteration (setting priority to critical).
Code reviews and issues
This description is plenty to get started with, so Rahul gets on with his development and testing, until he is ready for a code review. When he is ready, he commits his code to the review area, with the following SVN message:

[ issue 515 ] Proposed changes for adding a UI control that can bind to boolean
data (renders a checkbox).

You can view this commit at r2919, and as you can see, Google Code provides a convenient link to all of the files involved in the commit. GC also provides tools for attaching comments to any line in those files.
This revision therefore becomes the location at which the review is conducted, perhaps involving many comments from many reviewers.
But note also that Rahul's reference to "issue 515" has become a link to the issue itself. This means that a reviewer has everything they need to start reviewing -- links to all of the files that have changed, a reference to the issue being addressed, and some tools that allow them to add comments to the files.
However, anyone viewing the issue directly would not necessarily know that it was being worked on, or what the progress was. So to keep things up-to-date, Rahul added the following comment to issue 515:

Code for such a UI control has been proposed for addition in r2919 and review
requested. Changing Status to InReview.

The issue is now not only marked as being InReview, but we also have a convenient link to the code review itself. If you were interested in this issue, you would now be able to add your comments to the review, try out the proposed code, and so on.
Committing the code to trunk
At some point the review will finish, and the code will be ready to go into the trunk. (There may be other reviews first, but at some point the cycle should end.)
In our example, Rahul commits his reviewed code to trunk at r2920, with the following message:

[ issue 515 ] UI control that can bind to boolean data. Committed after favorable
review of r2919, with suggested filename change.

Note that we still have a reference to the issue number, but most importantly, we have a reference to the code review at r2919. This means that anyone looking at the code that is committed, and who wanted to understand why a particular change was made, could find the history of the discussion in the code review.
But we're not finished yet.
If we go back to the code review at r2919, we'll find that the last comment Rahul has added says:

Committed to trunk in r2920.

Now, anyone who stumbles across the code review directly knows that the code they are looking at really did find it's way into the trunk, and wasn't left hanging there uncommitted, or was abandoned in favour of some other solution.
Closing the issue
The final step is of course to close off the issue, and here it's important to refer to the revision at which the code was committed. In our example, the last comment on issue 515 is this:

Committed to trunk in r2920 after favorable review.

Resolving issue as Fixed.

Now anyone interested specifically in the issue can see not only that it is closed, but also where the code is that addressed the issue.
Conclusion
This technique may seem a little laborious, but it actually takes longer to describe than to do.
And by following these steps we ensure that we know how a sequence of comments ends, whether in an issue or code review.
We need to know how an issue was addressed when it gets marked as fixed, we need to know what happens to code after a positive code review, we need to know where some code came from that is committed to trunk; in short, no matter whether you are looking at one of committed code, an issue, or a code review, these procedures ensure that you can easily find the other two.

xforms

groups