My comment was “To Save Production Environment......We need a better Test Environment, ideally a mirror image of Production Environment.” I think one of the ideas of having a Testing team; is to Save the Production Environment from any failure whether it is functional or non-functional. There may be many points to counter the above statement but let’s agree that organizations spend a huge sum of money to be rest assured that the application will not cause any trouble once it is tested and deployed to end users.
What are the probable causes of Prod incidents?
No application works on its own. There are always some downstream and upstream apps as well. The incidents that I had experienced were a result of one or more points as mentioned below.
1) Two interdependent applications are being tested separately in different environments (for e.g. QA and UAT), both works fine but when deployed to prod, all hell break loose.
2) QA environment may not have the similar data size as in prod environment leading to performance issues specially when there are jobs running & supplying data to another job.
3) An Urgent requirement change is deployed to prod assuming it would not affect anything else and there is no need to test. The change was urgent because business was getting impacted. Now, after deployment the business is severely getting impacted because of an untested minor code change.
There can be many more reasons for failure of application in the prod environment. The last one is the most frequent one for me.
What I meant by asking for a better test environment?
Environment where all interdependent applications work the same way (in terms of data flow, data size) as they supposed to work in prod.
I have faced situations where an application was tested in test environment, no issues found in UAT environment but when the application was released in Prod, there were some serious issues. When the Root Cause Analysis was done; it was found that when application was tested in QA or UAT; it wasn't receiving data from one of the applications, so integration was never tested. But in prod; the application started receiving data and BOOM. The problem as I see it was the communication gap between teams involved.
An environment where the dataset and user base are similar to that is supposed be there in Prod.
An application broke in Prod because the data files it used to receive in Test environment were of size few KBs but once in Prod it had to receive files of size of MBs and it couldn’t handle the load. Kedar Kulkarni, a friend whose expertise is in Performance Testing, would agree to this.
In general, a test environment is shared by many applications so timely availability to different teams becomes an issue. In such cases, environment management becomes an issue. And it’s an irony that people, who want their prod environment healthy, don’t pay enough attention to their test environment which can be disastrous.
Can a better Test Environment Save the Prod Environment?
My friend Parthiban disagreed with my comment as mentioned in the beginning of the post. His idea is to have a stable test environment and then mirror the same to prod. Initially I didn’t like or probably understood this thought. Let’s evaluate;
If we can have all the interdependent applications stabilized in the test environment in terms of data dependency, data flow and data size than yes we can mirror it to Prod. If we can remove the problem of communication which is generally a result of ego clashes, yes we can.
I would request readers to share their views on the same and let's learn from each other’s experiences.
Hi Rahul,
ReplyDeleteI think one of the ideas of having a Testing team; is to Save the Production Environment from any failure whether it is functional or non-functional.
NO testing team/tester can save any environment; only a developer/environment manager could that too not on all situations. If a testing team is employed to *Save the testing environment* or for that that matter a software, the employee and the employer are in a trap and the sooner they get out, the better they would be.
On what we can do; we can find out possible reasons why an environment can go down and based on the additional competency, can find the root cause of those reasons.
My thoughts on Environment
Environment setup, management and maintenance is a niche competency (as testing is) and setting forth a clear expectation from it and having a test strategy is as important as testing. Because as you rightly pointed (I remember Pradeep telling me during one of our conversations), no software works on its own. It talks to lot of other software (including the OS) and hence the behavior of the AUT is hugely compromised on the quality of the rest of the software it interfaces with.
Having a suite an upstream and downstream application is probably a classic structure of any financial application and invariably true with capital market applications (from what I have seen). I agree with you that testing the applications End-to-End is as important as doing functional testing of application, because if the application is (arguably) bug free, there is no guarantee that it isn’t going to break when run with the lot
In such a situation as above, if an application was not tested in a cohesive environment with that entire component with which it would interact in PROD, the test strategy should be re-visited, better late than never, it’s the just right time set up such an environment.
I live with such applications day in day out, ie there are a suite of upstream application and a suite of downstream, written in different technologies in different architectures. When an order management system was build and moved to PROD, we also built a test environment what we call IE (Integrated Environment), and all our regression test instances include a End-to-End trade flow testing which is a complete cleansing of the system with a set of trades.
With all the above, re-creating a test environment as a mirror of PROD is impossible and emulating all the scenarios the application would be used in PROD by testers is highly impossible
There are a lot of reasons why I say the above
• With all the above, re-creating a test environment as a mirror of PROD is impossible
o Economically it may not be possible/feasibly to have the same hardware as PROD
o Maintenance of the environment is as costly as creation
o In Prod, the application could be reading from/writing into third party data feeds/sources. Connecting such data sources in TEST environment is unthinkable
o Data Security – mirroring the TEST environment means, not just hardware, the most important aspect, DATA. We have been seeing increased regulations and compliance checks around the way all NON PROD environments are handling the data
The list could just grow if we hear from people working in different domains/stream/technologies
@BugMagnet,
ReplyDeleteThink of how mobile manufacturers test what they do. Still do we face problems while same mobile phone/s used by 'n' users every day?
Think of how the automobiles are tested. Still do we face problems using same automobiles by 'n' users every day?
Think of how an aircraft is tested. Still do pilots or staff maintaining the aircraft's face problems using it everyday?
Think of how a trainee being educated by a person in training period. Does candidate educated in training period face problems in production or billed project or work?