This post is for early stage startups who have only a production tier and do their testing on their development environment. We have a short checklist of things to consider, some suggestions on evolving your test tiers and finally some advice on dealing with data.

---

A checklist for setting up your testing tier

A well designed test tier will efficiently increase your testing scope and decrease the testing cycles you need to regularly deploy a solid product. For early stage startups, we recommend keeping things simple. Have a tier that (almost) mimics your production tier. A checklist of things to consider:

• make it accessible to all employees.
• make regular deploys to the test tier easy for everyone.
• prefix ‘qa’ or ‘test’ to the application’s URL for clarity.
• add a (environment based) visual indicator in the application to remind users that they are on the test tier. It helps folk who use both the test tier and production.
• disallow (via robots.txt) search engines from crawling the test tier.
• redirect all outgoing emails/notifications to actual users to go elsewhere.
• consider copying production data onto the test tier regularly
• for data heavy applications, consider providing an ‘impersonate’ feature. Select employees can login as any user in your system. This feature is useful when trying to recreate user reported issues.
• If you have a copy of production data on your test tier, make sure sensitive data is masked and that you are not breaching agreements you have in place.

---

Other environments to consider

Once you have this much going, here are other environments you can consider depending on your needs:

• multiple test environments help different groups test different features without stepping on each others toes
• an environment for user acceptance testing that could be performed by business users/testers
• an isolated performance testing environment
• staging and demo environments

The downside of multiple environments is the cost and maintenance overhead needed to keep all of them in synch with one another. So unless you have great systems engineers in place, be extremely wary of increasing the number of test environments you have.

---

Having a copy of production data on your testing tier

For early stage startups, we would strongly advocate having a copy of production data moved over to the test tier regularly. Just do not fall into the bad habit of over-relying on client data to do all your testing. Benefits of having copy of production data for testing are:

• quickly replicate and troubleshoot customer reported issues
• helps you try workarounds on the testing tier without affecting your users
• better scenarios coverage quickly
• saves time when you need large volumes of test data
• useful for performance test execution with realistic/natural test data volume

Caution while using prod data for testing:

• sensitive data like SSN should be masked appropriately before loading into QA database
• thoroughly check data security terms & conditions
• ensure you are not breaching data security of your clients

---

Finally, a recent experience to drive home the point about regularly copying over production data to the test tier. I was booking an air ticket through a web portal. I entered all my details and submitted the form. My expectation was:

• payment should be processed successfully
• ticket should be generated with PNR ID

On the contrary, the actual results observed were

• payment was processed successfully



• but the ticket was generated without PNR ID

Technical customer care team ultimately took care of above bug. They generated a PNR through their backend. The explanation given was that this was a corner case and the feature was working as expected for others. The above case apparently occurred because the ticket price rose at the same time when the server was trying to generate the PNR.

Conclusion:  From a testing perspective, this kind of race conditions are hard to recreate. By having production data synch with the testing environment on a daily basis, this kind of corner case could be identified and included as part of testing for regression too