Dan Bricklin's Web Site: www.bricklin.com
Dealing with Disconnected Operation in a Mobile Business Application: Issues and Techniques for Supporting Offline Usage
Exploring the needs of one of the top concerns relating to business use of mobile devices




I've been writing here in depth about some of the major issues related to business mobile apps. Last year I posted my "An Overview of HTML 5, PhoneGap, and Mobile Apps" essay. It explained the evolution of HTML into HTML5, and helped define various terms like "wrapped", "hybrid", "framework", and "web app". A few months later, I posted "Responsive Web App Design" to explore the issue of better supporting the wide variety of different mobile device configurations.

This essay addresses one of the top concerns relating to business use of mobile devices: Operation when disconnected from the network. While these all come out of the work I do as CTO of Alpha Software to better understand the environment in which we are working, they are general essays not specific to Alpha's products.

Feel free to skim the headings and bold text for a quick introduction, however the details are important.


Overview
This essay covers some of the issues relating to mobile business apps and the needs of offline operation. It starts by explaining the evolution of mobile use in business and the growing importance of supporting disconnected operation. It goes into some of the ways in which this is different than the modes of operation most developers are used to and how needing to support it may be slowing the exploitation of mobile by businesses.

The bulk of the essay is about some of the different issues that must be dealt with when creating an offline-capable app. This includes maintaining the data on the device until a connection is reestablished.

A major section is devoted to the issue of getting the mobile transactions back to the corporate database. Many people incorrectly think this is a "solved" problem with common libraries that will do it automatically. Two different techniques are described, and their strengths and weaknesses explored. There is no single, simple answer, so understanding this should help in evaluating potential designs. The issue of database integrity and the difficulty of dealing with error conditions is included.

Mobile connectivity is often different than found on the desktop or normally in the office. This is explored and some of the implications noted.

Finally, there are user interface and user experience design issues that must be addressed. These include dealing with errors and keeping the user appropriately informed.

Links to some other resources are included at the end.

Addressing offline usage is one of the keys to mobile growth in business
After being pushed to have a "mobile presence" in the app stores, much as they were years ago to have a presence on the web, enterprises are now confronted with how they will start taking advantage of mobile devices for internal use. This mirrors the evolution of the "public web presence" followed by the internal web (the Intranet) that evolved from providing a simple phone book to actual dashboard and transactional applications. Native code "client/server" installed desktop applications were superseded in new implementations by web-based applications created with systems like ColdFusion, Perl, PHP, Ruby, and Java. The setup was similar: A personal computer directly connected to a LAN or remotely (dialup, Wifi, VPN, etc.). For quick implementation and easy deployment, this worked out well.

With mobile, however, the switch for internal applications has barely started. Research company reports support this statement. For example, the IDC 2014 Mobile Enterprise Software survey, which was fielded to US and UK-based organizations with 1,000 or more employees, indicated that 61% of organizations have deployed 3 or fewer applications, with 44% having deployed 2 or fewer. IDC analyst John Jackson feels that this clearly supports the idea that even in mature mobile markets a majority of enterprises are only modestly mobilized. "They also appear to be underachieving relative to their aspirations. We have other data that shows an intention to mobilize most business processes, but the matter remains that most enterprises are in the early stages or are stuck somewhere near the beginning of the mobile maturity curve." With so few deployed apps there has not been much experience to develop best practices.

With so few, you can guess that most of the custom-written apps they have deployed are customer-facing. This is in contrast to many existing large internal applications as well as a huge number of smaller ones developed using development tools like Microsoft Access or Excel/VBA, or the web-based tools. There are large corporations that have hundreds, thousands, and even tens of thousands of Access applications in use (the number of spreadsheet applications probably dwarfs that). All of these are targeted at non-mobile devices, with mobile devices having no access or only kludgy web support. Most of these in-house developers are not skilled in the latest computer languages and techniques (though they probably have a familiarity with HTML and perhaps a bit of JavaScript). They are, however, often skilled at understanding employee needs (at least in their area of the business) and database organization. They are comfortable with environments like Access, Visual Basic, the old Filemaker and Alpha Four, and their ilk.

One issue especially important to mobile device use that appears to be slowing down development and deployment of internal mobile applications is the need to support frequent or intermittent offline operation. Unlike a computer sitting on a desk used either in the office or at a desk at home, the connectivity situation for mobile devices is often very fluid and often unrelated to where it is needed. It is foreign to the situations encountered by other development and deployment environments.

Also, unlike customer-facing mobile applications ("B2C" and "B2B") that are often "must haves" with marketing dollars to help pay for experienced outside development, or diverted from other marketing projects that are lessening in importance or being superseded by mobile, many of these internal applications ("B2E" -- employee -- or "blue-collar" apps) require expertise within the enterprise and have much lower budgets and tight schedules. Digital agencies or young developers skilled at making an app stand out in the app stores and appeal to unmotivated potential customers, and focused on the latest client-side technology that supports users independent of each other with other support channels (such as telephone) available, may not be as well suited to dealing with the needs of the warehouse worker connecting to huge Oracle databases whose data integrity is of the utmost importance, and the fees the agencies charge may be out of range for departmental applications targeted at only dozens or hundreds of users.

Developers skilled in dealing with databases or desktop web pages are common within corporations, but those who can also develop for mobile are much less common and very hard to hire. Changing the template design of a web page to "responsively" fit on a mobile device and have a little animation is quite different than dealing with other challenges of mobile technology.

Unlike traditional browser-based business applications that are always connected to a server and the databases accessed through it, mobile applications frequently have to operate without a network connection and without the ability to immediately update or query the database. When the "system is down" in the office (hopefully an unusual occurrence), work often grinds to a standstill. When a salesperson visits a customer, they must be able to take an order even if it's in a "dead zone" -- that's a normal situation. For example, a salesperson may start a day while they are at home and have a Wifi connection by loading into their tablet information about all of the sales calls they plan to do for that day as well as product and inventory data. During and after each sales call, they query information about the customer and enter orders and other changes. Sometimes they will be in range of immediate data communications when they make the changes and sometimes they will be disconnected from cellular and Wifi data.

This is just one of many scenarios that require operating offline. Others include Wifi dead spots in a warehouse, lack of reliable connectivity while doing repairs in the field, poor connectivity in mass transit, and many more.

At Alpha Software, anecdotally, we are finding that the need for support of offline operation is much more common than most developers expect when they first start considering or prototyping mobile applications. The real world is not simple and consistent.

Corporate developers are finding that a large percentage of the mobile applications they want to develop will require offline operation at least some of the time. Coming out of the "always connected" world of the desktop in the cubicle, this is a tough barrier that often doesn't show up until a real deployment is attempted. Simulators and desktop office Wifi testing and demoing often give false security to the developer who is trying to make a simple prototype for "proof of concept". The delay and extra development cost offline support could entail at that point can cause project failure or hesitation to deploy another application.

The IDC data shows that a large number of the currently deployed mobile apps are "mobile web or browser based" (as opposed to hybrid or native) which are much less likely to support offline use. They also have a large interest in accessing and sharing documents and in-office use. However, some of the top issues they are concerned about are "issues in linking mobile platforms to existing databases" and they are targeting field service and salesforce automation. This speaks to a movement to more upcoming apps that will require offline support. They are also concerned that deployment took too long and that there were cost overruns. Widespread adoption will need appropriate costs and deployment speed.

The purpose of this essay is to explore the needs and techniques related to disconnected operation.

Issues
Here are some of the issues:

Persisting local data including unsynchronized transactions and app state
Getting mobile transactions back to the corporate database
Dealing with conflicts that arise from delayed updating of that database
Dealing with mobile wireless connectivity
Making all this clear to the user where needed

What follows is some detailed discussion about these issues.

Persisting local data including unsynchronized transactions and app state
The first requirement for disconnected operation is the preservation of data between the time it is gathered and when it communicated to the server. This includes being able to maintain it through time, through interruptions in running the app (such as to make a phone call, consult a mapping app, or show a presentation to a customer), as well as situations like the battery running out. A connected app often does not have as strict requirements here, since the time between data entry and communication is very short.

In many applications, and especially in business ones, there is often a need to keep the persisted data secure from unauthorized access. This includes when the device is stolen. Various encryption techniques are needed. This is a whole area of its own, and is usually included as part of the mobile device management (MDM) space.

Another requirement is for the app itself to persist. A normal web page, for example, usually needs to be reloaded from a server, even if intermediary data is persisted in temporary storage, such as using cookies or other means. There is browser functionality that can be used to cache apps on the device, much as images and other files are often kept, however this requires some special programming on the web pages. For installed applications this is not an issue, but installed applications sometimes need to go through app stores that can make updates hard to schedule or do on short notice.

A final type of persistence is returning the user interface to the proper state when an app is restarted. For example, partially filled out forms on sub-screens may need to reappear. Programming this capability can be complex, and many common consumer applications do not provide it. However, for data entry applications in business it is often a requirement.

Getting mobile transactions back to the corporate database
As explained above, when transactions occur and data is collected (either new data or changes to old data) information about those transactions must be maintained on the mobile device. At some point, after appropriate connectivity is available, those transactions must make their way back to the corporate database. For simplicity sake, and because it is a very common case, we'll assume that the data is going to a traditional database, such as MySQL, Oracle, or SQL Server, just as it would with an older, always-connected, web-based application.

Different strategies are used for architecting application software to facilitate getting the transactions to the corporate database, each with their own advantages and disadvantages. We will examine two of these strategies: Local Database Synchronization and Deferred Updating.

The first common strategy is to keep a replica of the database system on the mobile device. The application can then be written in the same manner as it would be if it was connected to a remote database over a constant network connection, but instead use the local database. Other software, part of the local and remote databases, takes care of automatically synchronizing the contents of the two at those times when a connection does exist. Most of the code needed to do the synchronization is separate from the application and need not be of much concern to the app developer. The same syncing database may be able to be used for multiple applications, depending upon the deployment system. An issue, though, is that the local database may need to have additional information about the data it is maintaining so that it can validate new data being stored and communicate validation errors back to the app at the time of data entry.

Conceptually this takes a big burden off of the mobile developer, especially a developer who is mainly used to the always-connected style of building applications. However, this is a double-edged sword: While the developer need not know much about how the synchronization techniques work, it also means that it is harder to know exactly what is going on and how to handle the complicated cases. Custom business software is often centered around dealing with special complicated cases (otherwise, why not use a generic system?). The developer can work as if things are always connected, and depend upon the "magic" of database synchronization. But, if the magic doesn't work, it's hard to understand why or under which circumstances. Lack of understanding can make it hard to predict such cases. For example, since the synchronization is not tied to an app, conflicts that arise when two people change the same field or other conditions that might prevent a change from making it to the main database may be hard for the user to associate with a particular form that they had filled out earlier in the day -- the database doesn't usually keep that information. Writing custom code for the database to handle these special cases may require the developer to be proficient in a different system and computer language than those used for the application itself. In the case of update collisions, some systems just use a simple algorithm, like "last-in wins", which may or may not be acceptable depending upon the situation.

Another issue with using an intermediate database arises when the database is in a different form than the corporate database. For example, the intermediate database may use a different variant of SQL or, in a common case, may be a totally different type of database, such as a NoSQL database. There may be the need to provide additional intermediate code on the server (or between servers) to synchronize the database that connects to the client with the backend corporate database. This adds additional work and complexity.

Yet another issue, in some cases, is that the local database is additional code that needs to be included with the app. Often, this code is native code, which requires the total application to be either native or at least hybrid. Using HTML5 for implementation and the web for deployment of applications is often a desired technique, which can preclude using such local native code. Some local databases are implemented using just JavaScript, though they often depend upon browser technologies that are not always well or widely supported, such as IndexDB. Over time this should change for the better.

Another issues with a local database is dealing with changes to the organization of the server database. There is also the issue of security, since in some cases the database may be separate from the app and need additional protection.

For brand new applications that do not have an existing database component, and that can work with the limitations that may exist with a particular server-side database, the local database synchronization technique may be better suited than in other cases. An example of a limitation might be the complexity of dealing with certain types of data integrity techniques when there are multiple people updating the same data fields or with updating multiple related parts of the database that may need to have possible rollback during a transaction. An application focused on logging discrete events or keeping a game's leaderboard, for example, may not be bothered by that limitation. An application focused on updating shared quantity levels of a scarce product may.

A different strategy for dealing with disconnected operation is to have the application itself keep track of database operations (such as creating or updating records). When data is needed from the database, or an operation is to be committed to the database, and if the connection is active, the database is accessed through the connection to the server and things work normally. If the connection is not active, then the application makes use of data that it has cached from the server and the application keeps track of changes to be made, buffering those changes up for future connections to the server. When the connection becomes active again, the deferred changes are sent to update the database through the server, and new data is downloaded if needed. We'll call this method "deferred updating".

This deferred updating method does not need a separate local database, and all of the operation is part of the application. While this makes what is going on more transparent to the developer, it also means that the application and the infrastructure supporting it has to keep track of all of the data changes and deal with the connecting and disconnecting. The application also has to deal with validation errors detected at the time when the data is finally uploaded.

An advantage of the deferred updating method is that the application can be built (and usually is built) with knowledge of the main database, and the server-side code that it communicates with can be built to know about the particular application. Status information determined on the server (such as data validation or collision information) can be tailored for sending back to the client, and the client code that communicates with the server can be integrated with the UI of the client app to appropriately display that status information and let the user act upon it as they determine.

A disadvantage of the deferred updating method is that there is a lot of extra, custom code that needs to be present in the application, as well as code on the server, including code to deal with error status. For hand-coded systems this can be a time-consuming, tedious, and error-prone addition to development, especially during early prototyping and concept testing, though for final deployment it could be the best solution. For no-code and low-code systems that build the code for you (on both the client and server) it is a more practical solution in all these cases.

An example of a situation that is complex to code is with transactions that affect multiple tables in a database. This could be customer information with different orders and different line items in each order. New orders need new identifiers to tie them to their line items, and there could be various conditions that are tested for on the server that could result in a rollback of some or all of the transaction. The user would need to be notified of the situation and possibly be able to remedy it.

Dealing with mobile wireless connectivity
In an office, wired connectivity, or even Wifi, is usually consistent at a given location (such as a desk). Bandwidth is high, especially with respect to back and forth simple transactions. On a mobile device, the situation is often quite different. Wifi can vary from place to place, even within a corporate office where IT has carefully balanced coverage. (Stairwells, elevators, and bathrooms may not rank with conference rooms or offices with widespread wired connections.) Cellular data is notorious for variability, even in large cities. High-speed LTE may be the norm in one place and much slower, early 3G not too far away. Latency with respect to multiple interactions with a server can be much longer than in wired or Wifi environments. Data plans can be expensive, especially when spread over many devices (e.g., a sales person with tablet, phone, and maybe even laptop) and roaming can add unexpected costs. Dropped connections are commonplace.

Handling offline transactions can exacerbate the situation. Long periods of being disconnected can lead to large numbers of data records to be updated. The amount of data can be much more than in a one-at-a-time situation, and coupled with the potential for low-speed this leads to the need to deal with long synchronization times. The long times, the higher incidence of sporadic communication failure, the need to pause or cancel an operation that is taking too long, and issues with latency may call for different communications organization than that which would be used for always-connected situations. Instead of sending everything at once and with almost no indication to the user (who just assumes everything is OK), appropriate batch size, and various handshaking protocols may be needed, with progress visible to the user.

Another issue is handling updates to the data that come from the server. In a traditional web application, the entire displayed dataset is updated with each transaction. In an offline situation, there may be a much larger amount of data that is cached, much of that perhaps not displayed except on demand. When a synchronization is performed, it is often desirable to update all of the information, including the extra cached data -- a two-way synchronization, not just sending changes from client to server. For bandwidth and other reasons, reloading all of the cached data may be impractical and only incremental changes should be transmitted. This may necessitate different data organization or other techniques not needed in a traditional web or desktop application. For example, the server may need to keep track of what data it thinks is on each device or the client will need to give the server some way to figure that out. There is also the user interface challenge of letting the user be aware of updates that may be of importance to them, much like "unread" in email.

Yet another issue with the realities of mobile connectivity is that you might need to do certain processing of data before sending it to the server, both to cut down on transmission time and to conserve local storage. For example, images may need to be scaled down from their full camera resolution to a smaller size that is still acceptable to the application. The synchronization of some data, such as images, may be separated from other, such as incident information, to wait for the availability of high-speed or less expensive connectivity for the data that can be deferred.

These are just some of the issues that arise from the switch to mobile connectivity.

UX Issues: Making all this clear to the user where needed
In most serious data-centric applications, loss of data is something that is to be avoided as best as possible. Even worse, though, is the loss of data without knowing that it occurred. Appropriately coded applications can protect the transactions that have taken place through a mobile device until the time when the remote data servers are updated. However, the user, if unaware of the fact that only their device has the information, could neglect to allow synchronization, take actions before synchronization that are contingent upon that synchronization or on its success, or, perhaps worse, delete the data from their device. When synchronization does occur, any errors, especially those that need to be dealt with in a timely fashion, and any updates to prior downloaded information, may need to be brought to the user's attention.

These issues have implications for user interface (UI) and user experience (UX) design. Appropriate status information needs to be worked into the display of data, and appropriate flow of use may need to take the needs of offline operation into account.

Email programs have always had to deal with such things. Historically, sending and retrieving email were "on demand" operations, and even are often today in the mobile environment. "Get Mail" buttons, "Draft", "Outbox", and "Sent" folders, "bounced" messages sent back to the Inbox, special "mail is being sent" sounds, and more, are provided to help the user understand what is going on.

In an app dealing with lists, for example, indications like you see in an email program may be appropriate for exposing unsychcronized data or submissions that are in error. Valuable screen space may need to be used for this. The ability to jump directly to a screen to correct an error long after the data was entered may be important. This is not something that is always built into an app when just capturing transactions.

To a programmer concerned about getting a working app built, these may seem like minor issues, but they can be key parts of the UI and UX design and need to be considered throughout the evolution of the development and not just stuck on at the end. They can require a lot of specialized code and reworking of tightly crafted screens if not part of the overall framework. Adding them at the end can result in a poor user interface or a large amount of recoding and delay in completion.

Conclusion
This area is very important to the successful adoption of mobile devices in business. I hope this essay helps add to the understanding of some of the issues and can help in the planning and crafting of successful applications.

Some other articles related to this that you might find of interest:
Alex Feyerke wrote "Designing Offline-First Web Apps" for the A List Apart web site late last year. He covers additional UI issues, such as how much the user should be aware of connectivity status and how being offline should be presented, some strategies for displaying updated data information, and some of the challenges in different types of apps. He also points to links that get you to a Github page for OfflineFirst / Research / Links related to offline-first. One of the blog posts listed there is mobile developer Aanand Prasad's "Offline Support is Valuable, and You Can't Add it Later" post from 2011. Another is Joe Lambert's "Offline First – A better HTML5 User Experience" from 2012 that walks you through a simple implementation of deferred updating.

Christian Heilmann, Principal Developer Evangelist of the Mozilla Developer Network, has a blog post from March 2014 which includes a 5 minute video showing "that HTML5 apps totally can work offline." This includes caching the web pages as well as the data.

In 2011, Sencha published tutorial material titled "Taking Sencha Touch Apps Offline" that goes through the code for caching an app and downloaded data (including images), as well as detecting connectivity. There is a section at the end discussing some of what needs to be done if you want to send updated data back to the server and the difficulty of programming it. A later article from August 2013, "Offline Techniques & Sencha Touch" lists various web technologies that can be used, but without the detailed coding in the earlier one.

Bob Moore of Alpha Software found Joel Grasmeyer's "A Comparison of Syncing Technologies" from October 2013. Joel was writing a new iOS app that needed to let employees sync data for sharing and backup, and iCloud was not appropriate for various reasons. He needed offline operation but was willing to look at solutions that kept entire copies of the database on each client: He calls it "replicate and sync" methodology. He points out the importance of being able to evolve the database design and migrating existing users (and their client devices) to the new versions. He explores the advantages of SQL-style databases in certain cases over NoSQL.

-Dan Bricklin, 15 July 2014

Michael Facemire of Forrester Research just released a report titled "Building High-Performance Mobile Experiences". From the promo material: "Performance is a key pillar of quality that mobile shops often overlook in their race to market. This report dives into the problem areas that impede mobile performance, and provides guidance on this not-so-black-and-white area of quality. " In the report: In the section "Offline Apps Are Great Performing Apps" he points out how hard it is to do offline yet how important it is do it anyway and explains why. Michael came to Forrester from IBM where he was in mobile software development and knows the difficulties firsthand.
17 July 2014

Here are some more links:

In a Twitter exchange, Xamarin's Miguel de Icaza pointed me to Zumero's Erik Sink who points to Couchbase's Chris Anderson's "Why mobile sync?" blog post from February 2013. Xamarin's Mike Bluestein pointed me to their 50 minute YouTube video of a July 2012 seminar titled "The Disconnected Mobile Device: How to keep your users happy in the face of slow and faulty network connections."

I saw a tweet from Christian Heilmann saying "Everything you need to know to create offline-first web apps" that pointed to front-end developer Guille Paz's "Offline-First Web Apps" Github page. This is an extensive list of blog posts, talks, and tools relating to handling offline with HTML5 apps.

In a discussion with Gartner Research vice-president Ray Valdes about this essay, Ray brought up the technique of "partitioning" an app so that parts of it may be obviously not in need to constant connectivity. For example, some parts could be obviously read-only, while the "needs synchronization" parts are clearly separated. In addition, there is the concept of "triage" when connectivity is missing, for example dropping back to doing only partial input validation locally (e.g., a zip code is the right number of digits) and saving more extensive checking for when there is connectivity.
20 August 2014

It was pointed out that the issues addressed in this essay are, in many ways, just as applicable to normal PC/laptop applications that need to work in a disconnected environment. In many ways, with the popularity of browser-only applications for a laptop (as opposed to the common use of installed native or hybrid apps on mobile smartphones and tablets) the challenges to doing a good job are greater.
3 September 2014

I posted a companion essay, "How Alpha Anywhere supports offline mobile apps" that goes through the issues raised here and shows how the new Version 3 of Alpha Anywhere addresses them.
7 September 2014
© Copyright 1999-2018 by Daniel Bricklin
All Rights Reserved.