Testing a New Product: Essential Information Software Testers Must Gather in the AI Era

  

 Learn what information software testers must gather before starting testing a new product, including AI, automation, compatibility, and release planning.

In my years of experience in software testing, I’ve observed a recurring challenge—incomplete product information at the start of testing. This issue is more prominent for freshers, junior testers, and newly appointed test leads.

With globally distributed teams, overlapping meetings, different time zones, and fragmented communication, testers/QA often miss critical updates. Important product details remain scattered across emails, tools, and meetings, making it difficult to get a single source of truth.

Based on my experience, I’ve compiled a comprehensive checklist of key questions every software tester should ask before starting testing on a new product, including AI and AI-agent–related considerations, which are becoming increasingly important in today’s software ecosystem.

Testing a New Product Guide

1. Product Documentation & Functional Understanding

• Is the functional specification of the product available?
• Are there user manuals, admin guides, videos, or hands-on documents?
• Is there a centralized knowledge base or wiki?
• If the product uses AI features, is there documentation explaining:
• AI behaviour
• Input/output expectations
• Limitations and known constraints?

2. New Features & Release Scope

• What are the new features introduced in the current release?

• Are any of these features: 

  AI-driven?
• Rule-based vs model-based?
• What is the target product version for this release?

3. Project Timelines & Milestones

• What are the key milestones?

• Feature Freeze
• System Integration (SI)
• Release Candidate (RC)
• Release to Market (RTM)
• EAR (w.r.t Mobile App release)
• GA (w.r.t Mobile App release)
• Canary Info (w.r.t Mobile App release)

• Are there AI model freeze dates separate from code freeze?
• Are there planned model re-training schedules?

4. Supported Platforms & Environments

• Supported Operating Systems (32-bit / 64-bit / Arm)?
• Supported Browsers and versions?
• Supported Mobile devices and OS versions?
• Supported languages (MUI)?  
• Does AI behaviour vary across platforms or locales?

5. Build, Deployment & Model Delivery

• Where is the application build located?
• Which branch should be used (main, master, development, staging, release)?

• How are builds delivered?

• Jenkins
• Testflight
• Internal repositories

• If AI is involved:

• Are models packaged with the build or deployed separately?
• Are model versions tracked and documented?

6. Agile & Work Tracking Tools

• Which tool is used to manage user stories and tasks?

• JIRA (on-prem instance or cloud instance)
• Azure DevOps
• AtTask
• Confluence (on-prem instance or cloud instance)
• Are AI stories clearly labeled (e.g., data change, model update)?
• Are test cases linked to AI acceptance criteria?

7. Upgrade, Downgrade & Compatibility Scenarios

• Is build-to-build upgrade supported?
• Is upgrade from previous versions supported?
• Is downgrade allowed?
• Compatibility questions:
• Can older clients work with a newer server?
• Can newer clients connect to older servers?

• For AI systems:
• Is model backward compatibility supported?
• How does the system behave if a model version changes?

8. High-Risk Areas & AI-Sensitive Modules

• Which modules need maximum testing effort?

• Are there AI components where:

• Output may vary for the same input?
• Decisions impact users directly?
• Are there confidence thresholds or fallback mechanisms?

9. Bug Reporting & Defect Management Guidelines

• Are there bug reporting standards?

• Important parameters to clarify:

• Found-in release
• Target release
• Severity definitions
• Developer or module owner contacts
• Default bug assignment to whom(Dev/Manager name)

• For AI-related bugs:
• Is this a data issue, model issue, or logic issue?
• Is the behaviour non-deterministic but acceptable?

10. Third-Party Integrations

• Does the product integrate with:

• External APIs
• AI services (e.g., NLP, Vision, Recommendation engines)?
• Are sandbox or mock services available?
• Are rate limits and API failures handled gracefully?

11. Test Automation & AI-Assisted Testing

• Are automation scripts available?
• Which tools or frameworks are used?

• Is the product suitable for:

• AI-based test case generation?
• Self-healing test automation?
• Are AI features testable via APIs or only via UI?

12. AI & AI Agent–Specific Questions Every Tester Should Ask

With the rise of AI agents and intelligent systems, testers must gather additional information:

AI Architecture & Behaviour

• Is the system using:

• Rule-based logic?
• Machine learning models?
• Autonomous AI agents?
• What decisions are made by AI vs human logic?
• Data & Training
• What data is used to train the model?
• Is test data synthetic or production-like?
• How often is the model retrained?

Explainability & Observability

• Can AI decisions be explained or logged?
• Are confidence scores or reasoning available?
• Is there an audit trail for AI actions?

Ethics, Bias & Compliance

• Are there checks for:

• Bias
• Fairness
• Hallucinations (for generative AI)?
• Is the system compliant with data privacy and security guidelines?
• Fallback & Safety Mechanisms
• What happens if AI fails or gives low confidence?
• Is there a manual override or fallback logic?
• Are guardrails defined for AI agents?

Why This Checklist Matters More Than Ever

Modern applications are no longer just rule-based—they are intelligent, adaptive, and data-driven. Without proper information gathering:

• AI bugs may go unnoticed
• Test results may appear inconsistent
• Critical risks may reach production

Early clarity ensures:

• Better test coverage
• Reduced ambiguity
• Faster onboarding
• Improved collaboration between QA, Dev, and Data teams

Final Thoughts

This checklist has helped me start testing new products with confidence, especially in projects involving AI and AI agents. While every product differs, asking the right questions at the right time makes a significant difference.

👉 I invite you to share your experience
What additional questions do you ask when testing software or AI-driven products? Please share your thoughts in the comments section.

 How to initialize your brand new project to Git using command - git init ?

Purpose: Start Git tracking in a project.

What it does:

●      Creates a .git folder inside your project

●      Enables version control

●      Does not upload anything online

When to use:

●      When starting a brand-new project

●      When you want to add version control to an existing project

Example:

git init

Terminal Output:

How to install Git ?

Steps to install GIT on your local system(Windows, Linux, Mac)

Before installing Git to your machine, it's a good idea to check if it's already installed onto your machine by simply typing the below command.

• Open a command prompt (cmd on Windows) or terminal(Linux / Mac)

• Type a command 

        $ sudo git --version

In case nothing shows (i.e. no output on the terminal) then you have to install the Git else you can update the Git incase version is of OLD.

Installation on Linux:

For RPM based distribution, such as CentOS or RHEL, use below command:

                $ sudo dnf install git-all

For Debian based distribution, such as Ubuntu, use below command:

        $ sudo apt install git-all

Installation on Mac:

MacOS comes with Git installed by default and you can check it via the terminal with command

         $ git version

Nevertheless, if you don't have Git installed for whatever reason, you can use Homebrew to install the latest version of Git.

Install homebrew if you don't already have it, then:

        $ sudo brew install git

Installation on Windows:

You can visit the official site to download the Git build/binary. Browse https://git-scm.com/download/win and the download based on your OS (x64 or Arm64) .

1. Once the installation has started by double clicking on the Git binary, follow the instructions as provided in the Git Setup wizard screen until the installation is complete.

2. Make sure to check the "Add to the system path" or similar option so that you can use Git seamlessly from anywhere in your system.

3. To verify Git is installed, open the windows command prompt and type $ git version

Reference screenshot :

What is Git ?

Git stands for Global Information Tracker. It is an open-source tool developed by Linus Torvalds in 2005.
It is a distributed version control system (DVCS) that allows multiple users to work on the same project simultaneously. It helps developers track code changes and maintain a history of modifications. Git is designed to handle everything from small to very large projects efficiently.

There are several features of Git, including:

Distributed workflow

With Git, developers can work on thousands of parallel branches simultaneously. 

Local repository

The Git repository stores your source code and its development history locally.

Collaboration

Git facilitates teamwork and innovation thanks to its powerful collaborative capabilities. 

Speed and efficiency

With Git, you can handle projects of any size quickly and efficiently

Robot Framework : Sample code to verify if Robot Framework is installed properly ? - part2

Sample code to verify if Robot Framework is installed properly ?

1. Create a file name test.robot and save it.

*** Test Cases *** 
Sample Test 
    Log     Hello from Robot Framework! 

2. Execute the script using below command. -> robot test.robot

3. Lets see this in details

a) *** Test Cases *** # This part is called Section and                                   its case-sensitive.

b) Sample Test # This is the user-defined Testcase name.

c) Log Hello from Robot Framework! # Here Log is inbuilt Keyword of RobotFramework for logging.                                           # Hello from Robot Framework! This is the message user want to log                                             during execution.

How to install Robot Framework on your Windows OS? - part1

How to install Robot Framework on your Windows OS ?

Robot Framework is an open-source automation framework mainly used for:

1. Test automation (acceptance testing, regression testing, system testing)
   
2. Robotic Process Automation (RPA)

It’s designed to be keyword-driven, meaning tests are written in a readable, almost natural language style using keywords that represent actions.

Steps to install Robot Framework :

### Open Command Prompt / terminal

        -    Press Windows + r key
        -    then type - cmd

### Check if you have Python installed or not, if not then install Python first

            python --version

### Create a folder and navigate to the created folder
          

       mkdir robot_f
       cd robot_f 

### Now create a virtual environment
            python -m venv robotenv

### Then activate the virtual environment
            .\robotenv\Scripts\activate

### Install Robot Framework
            pip install robotframework

### Check the version of Robot Framework
           robot --version

If you can see the robot framework version then it installed successfully.