Every project, sooner or later, ends up with accepting delivery at customer’s premises. The sooner the better.
International Software Testing Qualifications Board (ISTQB) defines: “In software testing the ISTQB defines acceptance as: formal testing with respect to user needs, requirements, and business processes conducted to determine whether a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system”. Acceptance testing is also known as User acceptance testing (UAT), End-user testing, Operational acceptance testing (OAT) or Field (acceptance) testing.
In short, User acceptance testing (UAT) is a process where the software end-users test the software in an environment almost equal to the production environment. The user verifies the software behaves as expected and that the requirements from the definition phase are met.
The UAT is critical phase of software development and needs to be performed as soon as there is a minimal set of functionality -ready to be tested. Often several testing sessions are required for larger projects. Sometimes the UAT can also be considered as integration testing as it shows the system behavior running in a different, more real-like, environment using the different set of resources as during the development.
Before the UAT is executed, it is crucial to define the exact set of criteria which must be satisfied for the customer to declare the testing successful. It is important that both the customer and the supplier understand that finding bugs during the UAT is highly acceptable. The system is still not ready for the production and every issue discovered during UAT can still be solved before going to the production environment. In my opinion, there are two major points which can make the UAT complicated:
at the UAT, the tester usually sees the software for the first time. To him, it might not look as he expected or he had a different idea in the beginning . This doesn’t necessary mean there is anything wrong with the product, it is only the level of expectation is what is different;
having a co-located test team or teams shifted across different time zones.
How do we deal with those at Amphinicy? Recently we carried on two testing sessions with customers in the States and in Europe. Both UATs were successful, but in both cases we saw examples of typical UAT at customer premises. The key of successful testing in both cases was in clearly defined test plan which was communicated to the customer prior to testing. Once the acceptance criteria was agreed, it was easy to do rehearsal before the testing. In that case, the unexpected issues could arise only because of difference in test execution environment.
The UAT session which was carried overseas was a bit more complicated due to the time zone difference. Email communication, although always the first choice, is often very difficult. When the team in the States started the testing, working day of the development team in Zagreb was almost over. A fix of a small defect in the product could have been delayed for a whole day. Proper planning and structured issue reporting was really of the highest importance. When the development team started with the fixes, they had to have a straightforward prioritized list of issues they should work on.
At the end, each UAT ended with a list of issues which could not have been resolved immediately. All the ambiguities in software requirements that got caught and all the defects which require bigger effort were marked for future discussion and development. Naturally, customer expected these to be fixed as soon as possible, preferably in the next release, to be considered as bugs and fixed free of charge. And, of course, without impact of the final release date.
The role of the UAT manager from the supplier’s side is to correctly and clearly separate the bugs from the change requests based on the original requirements specification. The UAT should not finish until everybody agrees on next steps. Summarizing the results of the UAT is the only way for successful and timely releases in the future.
Every UAT is a new challenge and both the customers and suppliers can learn from each and every one of them. “One learns by doing a thing (Sophocles)”. I would add: “And by documenting”. From software supplier side, communicating with a customer is a key to success.
We look forward to doing another UAT!