Since last few years, I have been working as a Test Engineer, testing applications of varying nature specifically in Finance domain. Moreover, my approach had been just like what I am told to do. For example; Write Test Cases (using some Test Management Tool), execute the written scripts and report bugs. At the end of day, send the Testing Status Report to stakeholders. Over the time I realized, “This way is not going to help me in longer run.” I had to do something to keep on learning and reporting some show stopper real bugs with an intention to make application better (less risky if not risk free) for the end users. So I started exploring things apart from my application like finding new ways to reduce cycle time (by means of Automation), report High Severity bugs etc and start questioning anything within my limits.
I asked that why we need to write Test Cases, when I know that this is not going to help. I knew that test cases are not covering all the functionalities because application is not a Web Based one; I am doing data validation and certainly cannot come up with generic test cases. I said Instead of wasting time on preparing Test Cases when I can put it in testing my application effectively.
Similarly, I asked questions on filling different kind of Test Metrics (DAR, CAR etc) which certainly do not convey any real usage. Someone replied that we use this data to benchmark ourselves with the best in the Industry and it helps us winning clients.
It still did not satisfy me completely because I knew that data is not always 100% correct. There is always some gut feeling involved. There is always some overlapping happening when we talk about SDLC or STLC. Everything is driven by money. In real life situation, money is allotted to project first and then the estimation is done whereas it should be other way round.
So, Why do companies follow conventional (Script based and so many processes) approach. This thinking lead to some questions-
Note: When I said Conventional, I meant following a plan in terms of writing test cases and doing script based testing all the time without getting much value, filling metrics etc.
Note: When I said Conventional, I meant following a plan in terms of writing test cases and doing script based testing all the time without getting much value, filling metrics etc.
Supporter of conventional methodology said;
1) How will you ensure that testing is complete if you don’t write test cases and mark their execution status by some means?
2) If after testing, a defect is found in Production (/Live) environment how would you say, “whether the underlying functionality was to be tested or there was some change in requirement which went untested?”
3) How would you measure Test Effectiveness in case you follow Exploratory Testing?
4) How would you measure and compare the performance of two testers as at the end of year, only one can be promoted or can get a better hike?
5) When one tester leaves an organization or project s/he working on, “How will you pass the knowledge if you don’t have things written? This is a risk area which has to take care of from company’s perspective.”
2) If after testing, a defect is found in Production (/Live) environment how would you say, “whether the underlying functionality was to be tested or there was some change in requirement which went untested?”
3) How would you measure Test Effectiveness in case you follow Exploratory Testing?
4) How would you measure and compare the performance of two testers as at the end of year, only one can be promoted or can get a better hike?
5) When one tester leaves an organization or project s/he working on, “How will you pass the knowledge if you don’t have things written? This is a risk area which has to take care of from company’s perspective.”
I could not find the answer to above-mentioned questions. I started reading some Slides made available by Cem Kaner (it was suggested by Pradeep from http://testertested.blogspot.com/ ) on Exploratory Testing. At one place, Cem mentioned "How to assess individual tester performance" under "Areas of Ongoing Concern”. This point acted as a trigger. I know Exploratory Testing is the way to go but still few questions remains.
I request readers of this post to please respond with your thoughts on the above listed points and help me learning.
I asked that why we need to write Test Cases, when I know that this is not going to help. I knew that test cases are not covering all the functionalities because application is not a Web Based one;
ReplyDeleteIf it had been other than web application, would test cases have covered all the functionalites in all scenarios and contexts in one go? Test Cases will be useful if it shows information. If it tells numbers count, then numbers can turn to be risk bombers any time. Numbers are Bombers!
I said Instead of wasting time on preparing Test Cases when I can put it in testing my application effectively.
What does 'effectively' means? It is very nearer for 'defectively' with little human error. For who ask test cases support them once you finish your work as you said "I can put it in testing my application". But going ahead go through this document once http://www.kaner.com/pdfs/GoodTest.pdf
. Search for other information also apart from the said PDF document. Nothing is useless and wasteful. Only the thing we don't know is how to make use them for our work.
Similarly, I asked questions on filling different kind of Test Metrics (DAR, CAR etc) which certainly do not convey any real usage. Someone replied that we use this data to benchmark ourselves with the best in the Industry and it helps us winning clients.
Probably we can not change everything in one day. I don't know what that DAR and CAR. If the client needs it, talk to them what kind of metrics they need and in what format they need it. Negotiate, but do not negotiate the value to the client. If this is not allowed for you do your work and also produce what they want at the end. For you it may not look like 'real usage' (what does 'real' and 'usage' means?). Example, for me wearing a spectacles is a needy one for daily life; but I don't for you, it may needy one you also, if not today may be tomorrow, or not at all. So do you object me from wearing spectacles. Finish your work and providing them what they need is a practice one should learn. And, do not provide information )or metrics what they call) with unrealistic and unanswerable data.
It still did not satisfy me completely because I knew that data is not always 100% correct.
Nothing is complete. That is why it is called data and you are not satisfied. If it had been complete and you were satisfied, probably me or you would not have got have been here for the day job we are doing.
There is always some overlapping happening when we talk about SDLC or STLC.
Everything is overlapping, if you see it like that. This is were we have role of finding information and try to understand what it can be. Any process outlines are just like a house or building having blocks and own usages. It is left to the individual in house how to make use of them or reconstruct them for their use.
Everything is driven by money. In real life situation, money is allotted to project first and then the estimation is done whereas it should be other way round.
Cost Vs Value.
So, Why do companies follow conventional (Script based and so many processes) approach. This thinking lead to some questions-
Demonstrate your ideas with result. Who knows they might also like your approach along with their approach also.
Supporter of conventional methodology said;
You are supporter of which methodology? There is not called best methodology or best model or best business practices or best industry practices. Also none said Exploratory Testing is best approach. But, it is turning to be a result oriented approach and practice along with other approaches that shows their own results (having their own importance with other approaches result).
1) How will you ensure that testing is complete if you don’t write test cases and mark their execution status by some means?
ReplyDeleteWho said testing is complete? I see that decision makers say take decision to conclude the testing here with information what you give to them for ongoing project context. Testing is never complete.
If after testing, a defect is found in Production (/Live) environment how would you say, “whether the underlying functionality was to be tested or there was some change in requirement which went untested?”
As said testing is never done or complete. You find always either it is in production environment word you said or development environment it was before going to production environment. Does all flower same fragrance. Who knows same flower might bring out the fragrance that you did not smell while it was in your geographical location. Let us work and practice such that we identify risks in the coverage areas. Risks, Bugs, Data, incomplete, learning (for few to mention here) are always there.
How would you measure Test Effectiveness in case you follow Exploratory Testing?
What kind of measure are they looking for you are thinking off. Quality or Quantity? Demonstrate with your results.
How would you measure and compare the performance of two testers as at the end of year, only one can be promoted or can get a better hike?
Rephrase this heuristic to "How can I help to tester to collaborate them with their learning.". If you are learning, you are promoted and getting a hike every moment. Why do you wait for the year end?
When one tester leaves an organization or project s/he working on, “How will you pass the knowledge if you don’t have things written? This is a risk area which has to take care of from company’s perspective.”
This is were document if did in way that can be understood by any one who reads it. Why do we put help document in the application that gets shipped? What happens if we put half boiled or cooked document into application that is shipped or will be shipped?
I remember and understand from Rapid Software Testing slide these below words:
The First Law of Documentation
That should be documented.
That should be documented if and when and how it serves our purposes..
If I am going out of the company, let the document prepared by me be helpful so that she or he can continue to add the value. Manage time, knowing Cost Vs Value, learning and practicing (only few mentioned) should all happen to explore ourselves.
@Learning,
ReplyDeleteI liked the way you have questioned things mentioned in my recent post. Probably I could be more diligent while penning down my thoughts, thanks for your time you invested here to share your thoughts. Well here are some more clarified points on your inputs;
"Test Cases will be useful if it shows information. If it tells numbers count, then numbers can turn to be risk bombers any time. Numbers are Bombers!"
Agreed, I am not against writing test cases. Term "Web application" was used for example, my observation is common to all kind of applications and if everything is client driven and iff client do not want "Test Cases" then should we write it even if it doen't cover most (if not all)of the features under test.
Testing Effectively - I am not an expert in testing but I can definitely say that if all the requirements along with acceptance criteria are in place and being followed too, number of bugs being reported after the release are within the pre-agreed (between client and vendor) limits then we can say Testing has been effective.
Cost V/S Value - Agreed a trade off has to be there.
"Demonstrate your ideas with results...."
Agreed, without data I can not claim that the approach being followed is wrong or right. All I meant to find a better way of doing things. I am not a supporter of any methodology just because there is no best methodology yet. As a tester, If learn more things and about more approaches then probably I can pick one to follow till I get a relatively better approach.
Moreover, I am not an independent tester, i work for a company and if company expect me to do things in a certain way then I'll have to.
Exploratory testing is "Simultaneous Planning & Execution". I read it somewhere on Cem Kaner's site.
But I could not understand how it can be done simultaneously. Help me with some examples in understanding this.
"Rephrase this heuristic to "How can I help to tester to collaborate them with their learning.". If you are learning, you are promoted and getting a hike every moment. Why do you wait for the year end?"
@- Hike in terms of Learning...ok its continuous, happens at every moment.But think from an Organization's perspective, if some manager says exactly this to his/her sub-ordinates and if the same does not reflect in terms of monetary benefits, would the person continue working in the same company? Would company like to loose resources?
One has to wait till year end because then only one get salary hike.
Note- @ Learning....Please share your profile publically so that one can read your thoughts as well and learn from the same.
Your comments are cordially welcome.
Thanks!!
@ Learning-
ReplyDelete"Probably we can not change everything in one day. I don't know what that DAR and CAR. If the client needs it, talk to them what kind of metrics they need and in what format they need it."
DAR - Defect Acceptance Ratio.
For Example, QA Team raised 100 defects and out of these 100 defects only 75 defects were accepted as Valid defects then DAR would be 75%.
Remaining 25% which were Rejected can be attributed to various reasons, like
1) Changes happened but not communicated to testing team
2) Tester did not have required functional/business understanding of the application etc. (the list is not all inclusive).
CAR- Causal Analysis and Resolution
It is a technique widely used in testing field (at least in my Organization) to find the Root Causes of a given situation. And situation can be anything like "Analysis of Reduced number of High Severity defects for a given application?"
This analysis requires some data and metrics serves the same purpose.
Also, saying there are no best practices, or standards is ok.....but whats the problem in having one. I see people ridiculing some standardizations (like CMMi) but do they have a better tried and trusted way to replace/answer it?
I ask questions because I want to learn and I don't see anything wrong with that. I am here to stay and learn.
Thanks!!!!
Rahul,
ReplyDelete"Test Cases will be useful if it shows information. If it tells numbers count, then numbers can turn to be risk bombers any time. Numbers are Bombers!"
Agreed, I am not against writing test cases. Term "Web application" was used for example, my observation is common to all kind of applications and if everything is client driven and iff client do not want "Test Cases" then should we write it even if it doen't cover most (if not all)of the features under test.
Try to answer this question yourself until you feel that you have got answer for that question. And if you wish to make a blog post, make it in your blog.
Testing Effectively - I am not an expert in testing but I can definitely say that if all the requirements along with acceptance criteria are in place and being followed too, number of bugs being reported after the release are within the pre-agreed (between client and vendor) limits then we can say Testing has been effective.
What is requirement?
How can I know all requirements that customer has made?
Does the customer has left something while saying the requirement (either in written format or said orally)?
Have I understood the requirements what the customer has said and is it the same when I infer when using the application?
If I have understood something else for the said requirement by the customer, does what I thought will also help customer?
Who has written the acceptance criteria?
Do I have any influence on the requirement and acceptance criteria?
Who will decide that this meets the entry criteria, exit criteria, acceptance criteria or any other words that books tells?
What if you and your team have advocated 10000 bugs? And the customer does not bother about this 10000 bugs. Then you report one more bug which becomes 10001 bugs that does not harm customer from using the application. Customer asks to ship the application. Where did the bug counts go?
After shipping the application you and your team continues to advocate much more bugs, which exceeds the pre-agreed limit. Now the bug count is 10701. Still the customer is happy, say though the pre-agreed defect count has been exceeded. What can we do here? Should we go on bug counts?
Pre-agreed counts or limit. Again the number plays here, If you advocate one information which costs the customer, does the pre-agreed count will be overseen? The value that your report says about the cost might revisit the pre-agreed count variable and might be it becomes preAgreedCount-- instead of preAgreedCount == n .
We (me also included along with you) should think of when we say testing is effective based on the bug counts, requiremnts delievered which was agreed upon, though it meets what you said acceptance criteria etc. As I said with little human error 'effective' is very closest to 'defective'. If you keep asking yourself, you will be answered by yourself for your questions you ask for yourself.
"Demonstrate your ideas with results...."
ReplyDeleteAgreed, without data I can not claim that the approach being followed is wrong or right. All I meant to find a better way of doing things. I am not a supporter of any methodology just because there is no best methodology yet. As a tester, If learn more things and about more approaches then probably I can pick one to follow till I get a relatively better approach.
Say you are walking in a place where molten lava is flowing on one side. On the other side you see the fire buring everything. You have only these two ways to follow for reaching the place you want go for. Which path do you follow till you get relatively better path (which was said by you). Do you make this path tailored for your moves now and give your best with what you have and learning further or you wait until you see relatively better appraoch (as you said) for your next moves.
Moreover, I am not an independent tester, i work for a company and if company expect me to do things in a certain way then I'll have to.
Yes, I understand you.
Exploratory testing is "Simultaneous Planning & Execution". I read it somewhere on Cem Kaner's site.
But I could not understand how it can be done simultaneously. Help me with some examples in understanding this.
I hope you can walk with your two legs. Stand inside the hot boiling water (100 degree centigrade plus temperature). Put your fingers into the boiling oil. If you can use your vehicle all alone, drive your vehicle against the busiest traffic direction in roads near your place. What would you do here? Will you be planning and executing your moves. Terrorist is standing front of you killing all the civilians. What will you do?
There are few exploratory testing videos of Pradeep Soundarajan. You should have seen them when if you visited testertested.blogspot.com provided there was no problem with whatever you saw from your end and the being viewed blog at the time when you saw.
Every human (tester) does the exploratory actions. .
"Rephrase this heuristic to "How can I help to tester to collaborate them with their learning.". If you are learning, you are promoted and getting a hike every moment. Why do you wait for the year end?"
ReplyDelete@- Hike in terms of Learning...ok its continuous, happens at every moment.But think from an Organization's perspective, if some manager says exactly this to his/her sub-ordinates and if the same does not reflect in terms of monetary benefits, would the person continue working in the same company? Would company like to loose resources?
One has to wait till year end because then only one get salary hike.
Hope you were thinking in terms of money, I was thinking in terms of learning. Sure when you are a manger you are not going do that. Believe your skills, learning, practices and understanding (which are always lighnting brighter and brighter when you keep looking and working for them) than the company. Then you dont even have to wait for a year for hike. I understand your words.
Also, saying there are no best practices, or standards is ok.....but whats the problem in having one. I see people ridiculing some standardizations (like CMMi) but do they have a better tried and trusted way to replace/answer it?
Try to question yourself why should I have best practice(s). Answer for yourself with your questions. Probably you have seen it as ridiculing that's all. Come out of that illusion in which you are if you think you are in it, else try to find in which illusion I am seeing this as ridiculing. If a person want a road to ride her or his two wheeler, should the road be of what kind so that they make use of it to travel in and out from their place. When I think of two different geographical locations have one approach (though nothing is best approach, practices or methodology) they find it easy to communicate between them (here communication is diversified, let us not think it as only exchange of what I should and take I have done.) though they might or might not understand their culture, language, lifestyle etc as I have understood.
I ask questions because I want to learn and I don't see anything wrong with that. I am here to stay and learn.
Asking questions, you are helping yourself to educate you. Analyze yourself whether yourself with what you are doing. If your writing as brought the QUESTIONS within you and the within the person who reads your questions, then might be your QUESTIONS ARE QUESTIONING.
I apologize. I have misspell the name of Pradeep Soundararajan above. Its my mistake.
ReplyDeleteRahul,
ReplyDeleteThanks for putting up your thoughts.
WOW....Thanks a lot for spending rather investing your time once again....I need to spend some more time to get back with my thoughts on your inputs.
ReplyDeleteI am wondering if you write blogs on testing. I am just curious to read your posts. If possible then make it public.Treat it as a request :)
Thanks a ton!!!!!!!
@ Learning... at least share your name.....
ReplyDelete@ Learning-
ReplyDeleteHike - Yes I was referring to money....though in terms of knowledge...we don't wait for year end...its a continuous process.
Best Practices- I do not understand this term completely probably "Better Practice" can fit well in current scenario. Also, I am not pointing at anyone or against any practice...it just occurred to me. I find it difficult to fill all these metrices and thats precisely the reason I want to understand its value (if it has any).
"As I said with little human error 'effective' is very closest to 'defective'."
I need to spend more time on this. Probably a good topic to write about...lets see how it turns out.
Thanks!!!! Learning.....
I find it difficult to fill all these metrices and thats precisely the reason I want to understand its value (if it has any).
ReplyDeleteIf one has to know the VALUE, then she or he if knows what it COSTS (if it has any) might help. I look for both COST(S) and VALUE(S).