web2.0

Yahoo! uses XForms for 'write once, run anywhere' mobile applications

Yahoo! recently announced their entry into the mobile web applications space, with a service that allows applications to be built and hosted using Blueprint, a purpose-built mark-up language based on XForms.

The Blueprint roadmap describes the underlying philosophy:

Much of Blueprint's philosophy and syntax comes from XForms. We opted for a full declarative language because it was the only way we could effectively run on the wide range of devices out there, some of which have no scripting at all. By using declarative syntax, we can encapsulate and hide the scripting specifics. In some cases, the code could run on the phone, in other case, such as XHTML, we can put the logic on our servers. It's the perfect way to deal with the various environments and their capabilities.

At the moment Blueprint is simply converted to XHTML plus JavaScript for delivery to devices, but the mention of XForms gives a clear indication of where they are heading. And given that Google did something similar with the launch of its Google Mashup Language, it all bodes well for a new wave of web applications that won't be built using JavaScript (or even Java and GWT) but will use straightforward, device-independent, mark-up.

Write once, run anywhere, anybody?

For more on why XForms is Ajax on steroids, see:

skimming at XML 2007 (and The Cloud's Silver Lining)

I've not yet had a chance to write up my experiences at XML 2007 in Boston last week, but an announcement today by Amazon prompts me to at least summarise one of the talks I did, called XForms, REST, XQuery...skimming. (Slides)

It's a theme I've touched on a little in this blog, and have spoken on before (at an XML UK meeting, last year, and at XTECH 2007, this year), and the idea goes something like this; building web applications by dynamically generating user interfaces that contain data, is awkward, and difficult to maintain. Web services (which includes REST and APP) removes some of the dependency on databases by hiding the data behind a simple abstraction layer--a kind of 'ODBC for the web'--and XQuery takes this further by giving us a standard language to get at the data.

But XForms is the icing on the cake, since it gives us a rich-client language that can be deployed like traditional web applications, yet is one that ensures a clean separation of the data and the user interface.

During the course of the presentation I showed how incredibly easy it is to create an application with XForms, since forms can be first created on the desktop--loading and saving their data from local XML files--before being deployed to any type of web server. By creating static forms that load their own data, rather than the more common model of dynamically generating static files that contain data, we end up with files that are completely agnostic about where they are delivered from. My demonstrations took them from the desktop, to a simple web server, to eXist (an XML database).

But these applications could even run in 'the cloud', with no web-server. That may sound odd, but I could store any of these XForms applications in SVN or Amazon's S3 and they would work as well as if they were in IIS or Apache. This is the ultimate in simple maintenance and future-proofing, since you are no longer reliant on any particular operating system or server-side language...the forms just 'are'.

GData

The final step of my presentation was to take this a little further and show an XForm that interacts with Google's GData; the form is simple, and accepts a longitude, latitude and comment which is then inserted as a row into a Google Spreadsheet, using Atom Publishing Protocol (APP). Another form is then used to retrieve the entire spreadsheet, and each row is used to create a pin on a map.

The great thing about this demonstration is that it shows that not only are our forms now agnostic about where they run--desktop, server-side, S3, etc.--but now the data becomes agnostic; we don't care how the data is stored, all we are concerned about is the protocol used to interact with it.

And crucailly, XForms' ability to get at this data shows that it really is the missing link when it comes to building a new kind of web application.

SimpleDB

But this generalisation about data just took another leap with the announcement today by Amazon of a closed beta called SimpleDB. This will allow programmers to store and query for data in a manner similar to the way that GData works.

By putting data into the cloud, and then using an easily deployed, declarative rich-client language like XForms, it is possible for the architecture of web applications to be altered in a quite fundamental way, in a direction that is completely different to what we're used to.

Sidewinder and the need for a semantic web applications framework

A couple of recent discussions in the RDFa and microformat communities concern areas of particular interest to those of us working on Sidewinder, a semantic web applications framework.

The initial discussion is taking place on the microformats lists, and concerns how to allow authors to indicate what actions are available to be performed on items appearing in a document. The second discussion is taking place on the 'RDF in XHTML Task Force' list; this post provides a good summary of some of the issues.

All in all I find these very exciting discussions, because they concern exactly the types of use-case that prompted me to get involved with the XHTML work at the W3C a number of years ago.

This was because I'd been trying to create the kind of flexible user interface that these threads are describing--no doubt just as lots of other people had--and in my own endeavours I ran up against a number of very serious problems that made me conclude that it was pretty much impossible with the technologies available at the time. And since I still haven't seen a convincing solution to the problem of creating extremely flexible user interfaces, I've concluded that the issues I ran up against are of quite a fundamental nature.

Some of the problems that seem to me to be absolutely necessary to solve are:

  • an HTML page contains insufficient metadata about what its content 'means', making it difficult to work out what kind of UI constructs to render;
  • even were you are able to work out what the data means, HTML is not itself powerful enough to express the kinds of complex user interfaces that you would want to 'bind' to this underlying data;
  • and even if you work out what the data means and define complex UI components, you still can't define binding rules that indicate what widget to use with what data;
  • the browser offers only one 'paradigm' for interacting with information of interest, whilst we often want to create applications that can make use of the same rich features.

For every problem...

The first problem--that HTML is not 'rich' enough--is now largely solved by RDFa. It has taken quite a long time to get here, but I think the effort that has gone into getting RDFa right is going to be worth it. The key thing that RDFa does is to get RDF into HTML--once you've got that, all sorts of possibilities open up.

The second problem--that it's difficult to define rich user interfaces with only HTML--is largely solved by XForms. That XForms is a solution is currently not obvious to most people so XForms remains peripheral in application development at the moment. Of course it is possible to define widgets using script but it quickly gets very messy. There is also an enormous problem of re-use, in the recursive sense; most Ajax libraries are works of art, but very few can support the kind of complexity we need. Take the example shown here:

Metabar showing metadata for a BBC news story

Here we have widgets that are made up of other widgets to an arbitrary depth, but the key thing is that the binding mechanism is based on abstractions; it could be the run-time data type or an abstract widget type, and in both cases it means that at any point in the hierarchy a different widget could be swapped in without disturbing anything above or below. This kind of complexity is extremely difficult to achieve with procedural languages. (See also Introduction to custom controls and Understanding the MVC separation.)

The third problem--the use of binding rules to indicate which widget should be connected to what data--is still a little in the air. One part of it we've solved by specifying binding 'rules' using XPath selectors that are data type aware. This means that we can indicate that data of type 'geo location' should have one set of behaviour bound to it, whilst data of type 'time' can have a different type. These are essentially binding stylesheets and I think this is where we can answer the question posed on the RDFa list as to whether it is the author, the end-user or the browser vendor that should be in control of the widgets--the answer is all of them! By allowing users to express binding rules that override those from authors, we can achieve the best of both worlds.

Finally, the problem that the only paradigm we have for interacting with web-based data is 'browsing' is what we are trying to solve with Sidewinder. The ultimate goal is to have a framework that can be used to build any type of internet-facing application, whether web or desktop based. By allowing all applications to make use of the same semantic functionality and the same binding rules we can allow the users themselves to get control of their data and their applications. (See also Web 2.0, Copernicus and Sparticus: Moving the centre of the web.)

Silverlight clock as a custom control in XForms/formsPlayer

Once you have this kind of 'platform' (see Platform 2.0) then things really start to open up. I could build a clock widget using Silverlight, perhaps, and have that appear anywhere that the time data type is used--whether in my messaging client, my email client, my Twitter gadget, my Facebook desktop notification system, and so on. But you might choose a clock built using SVG, whilst someone else might decide to have a clock that speaks. But any of us could swap one of these behaviours for another at will, for a component we have created or one that we have downloaded from elsewhere--the ultimate, flexible, programming platform.

Ajax and progressive browser enhancement

Progressive enhancement is an approach to web development that has been around for a few years now. (More on PE at Wikipedia.) At its most basic it suggests building our web pages in a 'clean' and uncluttered way, and then layering functionality onto the mark-up using various mechanisms, such as stylesheets and scripts. The idea is that browsers without the additional capabilities can still render the page in some fashion, but those with additional power--such as scripting support--can add more enhanced features.

This principle actually underpins much of the work we've been doing on XHTML 2, which has involved taking HTML back to its semantic roots. In particular the work on RDFa has been to a large extent motivated by providing more semantic 'hooks' on which to attach increasingly focused functionality. (The work pre-dates the term progressive enhancement, so it isn't generally called that.)

But there is a new phenomena afoot, which I'd like to call progressive browser enhancement. It's something we've been doing for a while with our work on XForms and formsPlayer, but it also has wider applicability.

DOM Events and standards

To illustrate the idea of PBE, let's look at eventing in the browser. Many Ajax libraries have their own eventing architecture, and they are without exception non-standard. This is a shame, since the W3C has a standard for DOM events (DOM 2 Events) which has been around for years and is very clearly defined. Of course, part of the problem is that whilst it has been implemented in Firefox, Safari and Opera, it's not available in Internet Explorer. This meant that when we began our work on formsPlayer, our XForms processor plug-in for Internet Explorer, we had to implement a DOM 2 Events component ourselves.

This does however mean that DOM 2 Events support is potentially available for all browsers, but of course the problem now is that in order to make use of it on IE, people would have to install formsPlayer. And even if we made the module available separately (which we will be doing shortly), the problem for the programmer is that they can't assume that it is installed; there is still an annoying rift between those who have browsers with DOM 2 Events support, and those that don't.

Using JavaScript to enhance the browser

To fill the gap we simply implemented a DOM 2 Events library in JavaScript. Whilst most Ajax libraries went the route of creating their own non-standard eventing architectures, we went the other way and implemented the standard. The advantage of our approach is that if our end-user is running a browser with reasonable standards support such as Firefox, Opera or Safari, they'll get native support for DOM 2 Events, and hopefully a faster experience. Similarly, if our user is on IE and has installed the formsPlayer DOM 2 Events component, they will likewise get a faster experience. It's only the middle group using the scripted version of DOM 2 Events who will have a slightly slower experience.

But the key point is that the end-user now has some control, since they can add a DOM 2 Event component if they want to--most likely as part of some other package--independent of the actual web applications they are using. And for the authors of web applications life is much easier, since they are coding to a standard, rather than being forced to code to the specific event architecture of YUI, Dojo, or whatever Ajax library they want to use.

This is the key idea of progressive browser enhancement.

Threading in Google Gears

Another example of PBE is threading in Google Gears. There are a number of JavaScript libraries around that create simple threading by using the timer in JavaScript. This is fine for a lot of uses but can affect performance. However, we can follow the same approach as I described above for DOM 2 Events; if the function calls that the programmer makes to create a thread are 'standard', then although in normal operation the less-efficient JavaScript version would be used, with no change to the code the web application can take advantage of 'proper' threading, should a user have Google Gears installed. (See also: Ajax makes browser choice irrelevant...but we still need standards.)

Of course, most users are unlikely to be looking for a threading library to install, just as they will not be downloading a DOM 2 Events module. But if these components are small enough to be bundled up with other pieces of software they will gradually find their way onto users machines as part of other, useful, packages.

However the distribution takes place, it's the end-users of our web applications that are in control of progressively enhancing their browser, not the web application programmers themselves.

These principles have underpinned the design of Yowl, a centralised notification system for web applications.

Yowl centralised notification system

Yowl is a way of providing notifications to users of web applications, but in a way that the user themselves can control. It was inspired by Growl, which runs on the Mac, and which is used by applications such as Skype and Adium; by passing all notifications through a central system users are given the power to decide which messages they are interested in, and how they want to see--or even hear--them.

Progressive browser enhancement allows Yowl notifications to make use of advanced features in the browser if they are available, but to fall back to basic notifications if not. For example, we've created a Windows component that allows messages to be displayed above the system tray. The component is invoked and called from script, and works with both Firefox and Internet Explorer:

Yowl notifications in IE with Sidewinder system tray component

However, if the systray component is not installed, the Yowl notification system will simply use JavaScript to show messages within the browser window. You can see a sample of the non-enhanced notifications here--it will run in all the major browsers:

Yowl notifications running in Firefox

The programmer need change nothing in their code to make use of the systray component if it is present, since as with the DOM 2 Events example, or Google Gears and threading, the user is in control of the browser enhancement, not the programmer.

(For more on Yowl see the Yowl open source project page, Yowl: Design principles and how to use it and Yowl: An open source centralised notification system.)

Where does this leave the browser?

There is still plenty of scope for the browser to evolve and make available the kind of features we're talking about here. But the truth is that progressive browser enhancement makes browser evolution far less significant than it has been until now. That's probably not a bad thing, given the chaos that is currently surrounding the development of HTML since the W3C gave its support to HTML 5; the spec itself is becoming increasingly top-heavy, and bears little relationship to the early principles of HTML. Far from having enormous 'kitchen-sink' specifications, PBE looks to add new features in a more nimble way, via small and focused modules.

Adriaan de Jonge on XForms and RoR

Adriaan de Jonge's article XForms vs. Ruby on Rails is a must-read for anyone interested in XForms.

You may not agree with all of it, but the author clearly understands the benefits of XForms, even if he ultimately comes down on the side of Ruby on Rails. And he makes some interesting suggestions as to where XForms should be going next.

The following quote is a useful summary of both the power of XForms, and the things we need to focus on to allow it to be used by more and more people:

XForms logic beats any other technique when used in XML documents with semi-structured nature. XForms logic is based on XML specifications and XPath queries. This notation requires a thorough understanding of XML, creativity with XPath, trial and error, and great talent for logical puzzles. For someone with the knowledge and experience of a software architect, the simple tools in XForms can be the building blocks of a very advanced and intelligent application.

Tech Talk at Google on Sidewinder

T. V. Raman, one of the key architects of XForms, invited me to do a Tech Talk at Google. The talk was on Monday, and looked at Sidewinder and our approach to using web languages to create desktop applications--XHTML, XForms, RDFa and so on. Although I really enjoyed doing the talk and having the opportunity to explain at length to a load of techies what exactly it is that we're doing, my overriding memory is already that of the lunch afterwards. Unfortunately, the lunch was not captured on video, although they do record the tech talks:

The abstract for the talk was as follows:

Web applications use HTTP to communicate with relevant services and manifest their user ... all ยป interface via HTML, CSS and JavaScript. With the advent of different gadget frameworks, they have finally broken free of the shackles of the Web browser to manifest themselves on the user's desktop as first-class productivity tools.

In this talk I'll describe Sidewinder, a framework for authoring and deploying web applications that are created as first-class desktop citizens. Applications created in this framework can not only communicate with the web, but with each other, turning the whole platform into a powerful tool for creating mashup applications. Sidewinder can also turn any other web application into a desktop application, such as GCal, KoolIM, Google Docs, and so on.

Another powerful aspect of the Sidewinder framework is the ability to display custom widgets based on the type of the data being processed--chosen dynamically at run-time. This kind of flexibility is crucial when building desktop mash-ups based on varying sources of web-based information, and Sidewinder goes further by making it easy to process the data-oriented web--including microformats and RDFa.

The presentation will include a number of rich internet applications that use the Sidewinder framework.

XForms is coming of age with version 1.1

A new draft of XForms 1.1 was made available last week. It's a last call working draft, which means that once comments made on the document have been taken into account, it will hopefully be possible to advance the specification through candidate recommendation, and on to a full recommendation.

To be brutally honest, in terms of features, XForms 1.1 is nothing to write home about. The bulk of the new stuff was needed long ago, and we could have done with having it in there before now: looping and conditional constructs are crucial to any language; the ability to set URLs and headers at run-time is a 'must-have' when using SOAP, WebDAV, ATOM, and REST; and being unable to insert nodes into a nodeset when all the nodes had been deleted was painful! Thankfully, those days are now behind us. :)

But just because none of these features are unanticipated, doesn't mean that this document is not an important one.

For a start, in many places it's clearer than XForms 1.0; although W3C documents are always a team effort, the improving coherence is due to a lot of hard work from John Boyer, as he's often prepared to go the extra mile when writing an explanation or giving additional examples. (A sense of what has changed can be seen by looking at the version with all differences highlighted.)

Secondly, whilst I might have a moan that the new features were slow coming, the point is that they indicate the commitment on the part of the working group to solve real world problems, and more importantly, they show how the underlying XForms architecture easily lends itself to the addition of new features in a natural way.

And finally, with the new submission features of version 1.1, and the already powerful XML-handling that was a hallmark of version 1.0, XForms becomes the obvious choice for quickly creating rich internet applications that use ATOM, WebDAV, and REST. That XForms does 'come of age' is crucial if, alongside the proprietary offerings of Adobe and Microsoft, there is to be an approach to building internet applications that is based purely on open standards.

New drafts of XML Events 2.0 and XHTML Modularisation 1.1

I've been working with Shane McCarron from the HTML Working Group on a number of different documents lately, and it's great to see them made publicly available recently. The first is a new draft of XML Events 2.0, and the second is a new version of XHTML Modularization.

XML Events 2.0

The new draft of XML Events has features that are specifically geared towards future versions of XForms and compound document mark-up.

Some of the interesting new features are:

  • the ability to register and remove handlers at run-time;
  • an attribute for executing events conditionally;
  • another for repeatedly executing actions whilst a condition is true;
  • the ability to create action handlers using script, which means that script can be interspersed with other action handlers.

XHTML Modularization

XHTML Modularization 1.1 (or M12N as is it is often called), is a set of guideines, DTDs and XML Schema modules that provide the basics of XHTML, ready to be recombined in different ways by language designers. An example of its use is XHTML Basic, aimed at mobile devices; this language is a sub-set of the full XHTML, and so uses only some of the modules provided in the M12N 'library'.

It's of particular interest to us at x-port, because we've been using compound document schemas for a few years now, and our work in this area has been fed into how these schemas are structured. We use a language created with M12N--we call it xH--that incorporates XForms, SVG, MathML and RDFa, both to drive the editing of our XForms documents, and to validate documents loaded into Sidewinder. The extension languages are defined as modules, following the principles of M12N, so they could be used in other languages, too.

Understanding the XForms dependency-engine

At the heart of an XForms processor is an incredibly powerful 'dependency-engine'. It works much like a spreadsheet, in that you tell it what calculations you want performed, and then the processor works out when to do them, based on changes that take place in the data in the form, as the user interacts with it. Although this all happens automatically--just as in a spreadsheet--knowing what's going on 'under the hood' will mean an XForms programmer can avoid writing inefficient code.

To help programmers gain a better understanding of this little-discussed part of XForms, I've added Understanding the XForms dependency-engine to our Programming with formsPlayer and XForms handbook.

Comments would be most welcome. :)

formsPlayer.com updated--the first place for XForms programmers

We've been working hard in the last few months on a number of different things. Some are imminent--such as full support for XForms 1.1 in formsPlayer--and some are almost imminent (and we can't talk about them yet). But one that is here now, is the fresh, new, restructured and redesigned formsPlayer.com site.

Thanks to an excellent Polish designer now working in Ireland, who was introduced to me by an Englishman who we worked with on a project for a Canadian customer, formsPlayer.com now has a new look. And I don't mind saying, I think it looks great...thanks Bartek!

Up-to-date

The main change in the site is that being based on Drupal, it is now much easier for us to update, which means that we can add tutorials, samples, articles and blogs whenever the mood takes us--and that's often. We've also moved our bug-tracking onto the site, and of course there are plenty of forums in which people can ask (and answer) questions, as well as share ideas.

Integrated

One of the main advantages of the new site is that everything is completely integrated; a comment in the bug-tracking system can link to a tutorial, a blog entry can link to sample code, an answer in a forum can link to a feature request. Being able to cross-reference everything means that the information and knowledge in formsPlayer.com, about building web applications with XForms, will get deeper and deeper.
With these changes we will continue to make formsPlayer.com the foremost place to go for XForms software, samples, tutorials, forums, and more.

An XForms toolbar for searching the Prototype API documentation

formsPlayer toolbar for searching Prototype API docs

A recent announcement that documentation for the Prototype Ajax library was now available was followed almost immediately by a rather neat looking desktop gadget and a Firefox sidebar. Not to be outdone, I decided to put together a toolbar for Internet Explorer, and managed to do the whole thing with one XForms form control and an action handler--and of course, no script.

If you use Prototype and you just want to install the documentation toolbar, you can do so from here. You'll need formsPlayer, but this form will install it automatically, and once you have it you'll be able to add other bars to IE without any further downloads.

I've written the whole thing up as a tutorial which might prove interesting even if you don't want to create any toolbars, as it shows different ways to use load, as well as DOMActivate on an input control. If you'd like to get the code and use it as a guide to creating your own toolbars, it is available from here.

When should I use XForms?

The question about how XForms should be sold to customers often comes up, and generally we reply "why would you?". As in any IT project you need to use the right tool for the job, so why would you try to sell XForms in and of itself? Sometimes XForms is the right thing, and sometimes not, but either way--we say to those who ask--it's a tool in your armoury (if you want to use it), not your customers'.

If you have a system that uses web services a lot (for example), then a rich client that can handle XForms will be ideal--that's where most of our use cases are coming from for formsPlayer deployment. I've mentioned before a system that we've built for a customer that involves using the OpenWFE workflow engine and the eXist XML database, and we've written practically no code on the server! All we've done is used a few workflow templates, and the REST/XQuery interface to eXist, and everything else is achieved via the XForms themselves. Maintaining such a system is not only very straightforward, but is easy to move to other platforms, as well as swap components in and out (since the interfaces--such as XQuery--are all standard). This is all at the heart of the skimming approach we've been talking about.

But unfortunately, at the other end of the spectrum, if you have a system that uses a server-side scripting language, with direct queries on a database to populate list-boxes (that whole nineties thing!) then you might not find XForms that useful. You'll have a maintenance nightmare with such an application, either way. For XForms to be of benefit in this scenario it would have to dramatically improve the user experience, rather than only being 'as good' as what you have already.

I'm not differentiating here between using client and server technologies, but rather between the use of high-level constructs versus painstakingly creating pages with server-side script. So in the approach I'm suggesting, you will also get productivity improvements if you can use a pipeline-based system like Orbeon, since it too uses 'big constructs', even though it delivers HTML to the user.

At least in the short term, the main benefits of XForms will generally accrue to the developers and not to the users, and will be of most use to those who have a service oriented architecture, rather than an old-fashioned web-site. So trying to 'sell XForms' is probably not only a waste of time, but actually unnecessary--if it fits your system, use it.

Using the XForms model without the user interface

The XForms model provides a rich set of features to manage data submission, calculation dependencies, validation, control notification, and so on. Obviously it's normally used with the XForms user interface--controls such as input and range, as well as structures like switch and case or repeat--but we've recently posted a couple of examples on skimstone where a non-XForms user interface is driven by an XForms model.

The first concerns a 3D visualisation, linked to a simple XML file that contains a list of parts. XForms is used to select parts of the visualisation depending on which item in the part-list is chosen.

The second example involves a standard Google Map, with each of the pins that are overlaid onto the map coming from XForms instance data. In the examples the data is an RSS feed that contains geo data, which means the code can be used to overlay any feed onto the map.

Although our preference is invariably to wrap these types of features into reusable components that are instantiated with XBL, it's still sometimes worth showing examples like this since they provide an interesting link between XForms and Ajax.