Papyrus Application Scalability

Many Papyrus Platform installations have recently grown substantially and therefore means of application scalability have become a common subject. The principal scalability of the Papyrus Platform is unlimited due to its peer-to-peer cluster design but it is restricted by the synchronicity needs of the application. Simply reading or displaying documents or content from storage has no limitation, but keeping multiple write accesses in sync creates scalability issues. Clearly, when the number of users is doubled from 1000 to 2000 users then it does require monitoring and if necessary scaling the HW. When the number of documents or process tasks is doubled from 1000 to 2000 per hour that also requires consideration ow that load can be safely spread across server nodes.

Scaling Papyrus Platform applications is very simple compared to for example three-tiered Web/Java/SQL/SOA application clusters. A more detailed discussion of Java Application scalability I have posted on my ‘Real World’ blog.

User complaints that the ‘application is slow’ without any details being measured are not helpful but have to be taken seriously. Often the user has no means to understand that the processing requirements for similar looking documents can differ substantially. The document might be simple for the user but the backend process can be fairly complex. We propose that proactive monitoring is established. The Papyrus Platform has all the means for such monitoring available.  Simple dashboards can be created and summary reports are available.

Scalability is not only about tuning or maintaining an acceptable response time for a growing number of users. It is unreasonable to expect that the system will handle growth in users and transactions automatically. No system does. Papyrus has many load balancing and tuning options and many are set either by default or by the system. We found that some automatic functions had to be changed for weak networks or when growth was not gradual but for example many users, nodes or new documents and processes were added at once. That is related to the automatic version control and deployment of the Papyrus Platform.

Document application are a complicated conglomerate of GUI, process, and rule execution threads that read and write data from a number of service interfaces or databases. The performance of the SOA backend service interfaces or the database is much more relevant to the scalability than the user front end. Using common database or transaction measurements the Papyrus Platform executes millions of transaction per hour. That is however irrelevant. The question is how that translates into user experience!

The following measurements are used to define the quality of scalability:

1) IRT – Initial Response Time: from request to first usable feedback

2) TRT – Total Response Time: from request to completed function

3) SRP – Service Processing Time: the time the request is actively being processed

4) SQT – Service Queuing Time: the time the request waits for processing

5) ATT – Average Transaction Throughput: transactions performed in a time window

These values can not be taken from the system itself but the measurement of user experience has to be defined in the application. In a multi-tiered Java application that is practically impossible. A screen refresh may or may not be connected to a previous user entry. Papyrus assigns a JOB-ID to each user data entry and enables the tracing through all the functions and servers. A JOB-ID will now trigger the five above measurement points and thus enable a real-world measurement. It is planned to provide a ‘User Experience Dashboard’ for each user that will display a kind of ‘VitalSigns’ statistics for real-time feedback.

The Papyrus Platform uses the Application Performance Analyzer or APA to measure those values and relate them to the general statistical data about CPU, I/O and RAM usage.

APA offers a unique level of insight across all application functions from the user click on the desktop or portal to the final display. Next to the elapsed time measurements, the measurement of resource usage is the key to understanding and tuning. How much CPU, RAM, disk I/O and network bandwidth is consumed per transaction and in total has to be known for tuning.

Web/Java/SOA applications have substantial overhead for sticky-load-balancing, transaction-safe Java caching and database clustering, and parsing and validating the XML data for SOA data communication. Rather than the immense complexity of clustering multi-layered caches – with multiple conversion from tables to cache pages to objects and reverse – the Papyrus Platform collapses the horizontal structures and work with a purely (vertical) object model from definition to storage. The proxy replication mechanism of the Papyrus Platform uses a partitioned caching concept where there is always a unique data owner and each server node uses a replicated copy that is either pull or push updated. The objects do not have to be de/serialized as they are cached/replicated/binary-stored as is. The same object caching mechanism works transparently for all objects regardless whether they are populated via Web services, external databases or from the Papyrus objectspace.

As a consequence of this future proof design the Papyrus Platform can scale linearly as servers and nodes are added to the application clusters. It is however important to segment the database properly. Papyrus provides transparent object access and search across any number of nodes without any changes to an application. it is important to understand that even with a powerful system such as Papyrus the application has to be defined scalable as much as the system.

Forrester Research rates Papyrus a ‘Strong Performer’!

I this post I want to comment a bit more on ‘The Forrester Wave – Document Output for Customer Communications Management’.

Overall, Analysts Sheri McLeish and Craig Le Clair set a fairly high goal to analyze the current market offering in document output management. Only vendors with enterprise class functionality in interactive, on-demand and structured document creation were analysed. The ISIS Papyrus Platform was positioned as a strong performer in the same league as all the large vendors. No overall leader emerged.

“ISIS Papyrus by far owns the broadest vision for DOCCM in the industry,” stated The Forrester Wave™: Document Output For Customer Communications Management (DOCCM), Q2 2009. Forrester defines DOCCM as software used to compose, format, personalize and distribute content to support physical and electronic customer communications and improve the customer experience.

The ISIS Papyrus Platform v7 was reviewed in The Forrester WaveTM as follows: “ISIS scored well as a Strong Performer across all segments and as a well-balanced product with enterprise potential. ISIS Papyrus has the broadest and most unique vision for DOCCM in the industry, supporting ECM, CRM, analytics, event processing, and BPM. It has a modular and easy-to-configure system based on patented technology that combines with innovative post composition, output management, and production management.”

Forrester also reviewed reference installations and stated: “Customers are committed to the ISIS vision and feel passionate about this vendor.”

I am pretty satisfied with the report but it obviously is highly subjective how the functionality, strategy and market presence is rated. If you purchase the report you can change the weightings in the spreadsheet to find your own leader. What comes out of these reports has a lot to do with how clever the vendor positiones features in relationship to buzzwords and trends at the analyst firm. As Papyrus is always against the current trend we usually do not too well in such reports except when a more detailed comparison and an in-depth evalutaion of long term benefits is made. Butler Group did that excellently.

Also at Forrester marketshare and business size is still number one. Therefore a fairly weak strategy and little innovation goes a long way with a huge vendor. A smaller vendor who is very innovative is seen at risk because of that. So you cannot expect that small innovative businesses are at the forefront in terms of ratings, except if once again you hit one of the buzzwords that matches the analysts predictions! There is kind of a commonality between IT and stock market analysts and it is related to self-fulfilling prophecies.

I noticed a lack of understanding in what the various product functions actually are supposed to do. The true complexity of structured output functionality was underrated with fairly basic documents being considered as the norm. For interactive output Forrester mostly rated forms filling applications that in most cases are not enough for the most basic requirements. The document generation with such vendors is then always additional hardcoding in Java or similar. OnDemand generation is defined as document request from other application and widely underestimates the need for interactive text editing as well. Do not forget that you also need the ability to collate, bundle, sort and resend those documents either on paper and electronically and that this ability is neatly integrated with the Papyrus Platform.

What Forrester considered as being ‘Workflow’ or ‘process’ for document application cannot really be taken into account. Only the large BPM setups of either EMC or Oracle would come close to what customers are asking and getting today from a Papyrus Platform. What Papyrus EYE offers in terms of GUI not even Adobe FLEX can come close, but how would Forrester understand that? Therefore if you need highly user-customizable content, free-text editing, dynamic processes for creation and production, powerful color charts, image and graphics you better make sure that you do a proof of concept before signing a contract.

Papyrus Project and Services Planning

Most Papyrus Platform projects go quite smoothly and deliver powerful functionality without the need for a complex development project. The broad functionality of the Papyrus Platform enables larger and more consolidated applications, with the consequence that some of the projects in large corporations are a struggle despite the advantage that there is no Java or other programming necessary and that the number of involved people is quite small.

Once Papyrus gets chosen as the solution it means that business users saw something in the demo that they liked in terms of frontend and IT saw the functionality required in terms of technology. Neither business nor IT seems to accept that there is a need to use their specific business data model and interfaces, create the role/policy setup for authorization, define the building block objects, setup the administration workflows, define and test the backend data interfaces (despite XML, it is always the last thing provided for testing by IT), interface the document creation with printing and archiving and some more linkage here and there. Once that has happened the business workflows have to be defined. We always recommend to use the UTA to train them, but often someone decides that the workflows have to be rigidly defined so that they can be audited in advance. So suddenly someone (usually a third party) sits down and does lengthy process analysis.

All the while we are typically training and supporting third-parties who have never seen the product before because they were chosen as the cheapest manpower option. Project management takes a lot of time as well with daily meetings that take away time from implementation. Project planning and timing hardly ever happens according to business needs and IT capabilities, but influenced by company-political turfwars and budget constraints. The later we are in the project, the more management gets involved who know nothing about it.

Then the document templates and components have to be defined by the business users. Instead focusing on that business users come forward with the first change requests to the user interface – that they originally liked. Not too difficult, but it is an uphill battle against time. New requirements appear daily and are made no-go critera without changing the productive date. The planned time for testing gets shorter and shorter and no one takes the time to define proper testcases and acceptance criteria. No people were trained to run production but the go-live date is adhered too. Obviously there are issues and therefore each and every situation is reported to ISIS as a BUG to get support for free. Often there are issues with other applications that we are connected too or network or server issues and these are ALL ‘Papyrus problems’.

Does that sound familiar? Yes, it can get very frustrating! Depsite all of that the team at ISIS Payprus has ALWAYS delivered. The projects that were not taken productive failed after the Papyrus implementation was complete. Look, I am not blaming anyone. I am just stating the facts. I cannot laugh about the Dilbert comics, because they describe corporate ignorance too accurately. But enough of black humor. Let’s look forward on how to avoid these situations. It is really not that difficult!

What can we and the business users at our customers do to speed up projects?

1. Create a milestone plan of implementation phases where each states the functional goal or achievement.
2. Project phases must not have a rigid completion date and be able to overlap with the next phase.
3. Utilize a business architecture team, one or more implementation/test team(s) and analysis/tuning team(s).
4. Business architecture should be inhouse IT, analysis/tuning from business and IT, implementation can be external.
5. Start with a simple phase that can produce benefits quickly and that allows gradual growth and expansion.
6. Papyrus is LEAN technology by design, so it makes sense to remain AGILE (expand iteratively) in approach.
7. Utilize teams in a mode that allows overlap of analysis, implementation, testing and tuning for phases.
8. ALWAYS COMPLETE the first phase without add-ons, but add new features to later phases.

Overall the project approach would look like this and enable the business users to get early results and be closely involved in tuning the application to the optimal functionality in the later phases.