White Box Testing
Test your code
   Home      Test-Distribution-With-Docker
 

Scalable test distribution, test independency and configurable retry with docker and friends

 

Unstable and slow UI automation in CICD

  •   Test executes in sequence and test dependency. This is the major cause of unstable UI automation If We run UI automation test in sequence on same web driver instantiation
 
  •   Slow in execution UI automation. If we run hundreds of test in sequence that takes hours  to execute. We can do test distribution with thread based approach (ex parallel gem for cucumber ruby) and that is non scalable
 
  •   Non configurable retry for number of attempts. If System Under Test unstable or test written more flaky we want to retry number of times
 
 
 

Stable and faster UI automation execution in CICD

 
  •   Run each test independently in sandbox environment. Instead of running test suite in sequence with single web driver instantiation, run test independently with web driver instantiation for each test 
 
  •   Test distribution on lightweight docker containers. Build docker images for required platform and run each test on docker container. Docker containers are light weight and depending upon computing device We can run hundreds of test in parallel. With headless browser and docker we can reduce execution time for regression from hours to minutes
 
  •   Configurable retry for failed test cases
 

Approach to implement this test infrastructure for Cucumber, Capybara with ruby


 

  • Configuration for environment, retry and tags to execute. Other approach to pass them as commandline parameter or environment variable

FLAG = '@regression'
APPENV = 'qa'
RETRYCOUNT = 3
CONTAINER = 'my_test'
  • Build docker image with Dockerfile in attached source

sudo docker build -t my_test .

  • Dry Run to filter test scenario for tags and profile. Update your test framework for before all hook to quit driver and raise exception, So that all test get fail immediately and we get list of test scenarios as failed test


# Dry run to get list of test scenario
output = `sudo docker run -a stdout -i #{CONTAINER} "cucumber -p selenium --tags #{FLAG}
          app_env=#{APPENV} dry_run=true -f rerun" | sed "s,x1B[[0-9;]*[a-zA-Z],,g"`
output.split(/ /).each{ |testcase|
$testcases[testcase] = RETRYCOUNT
}
$logger.info("Total Test Scenario #{$testcases.length}")

  • Credential pool to run each test with different credentials, So that test don’t clash. This required in case application requires loginlogout credentials. 


# Populate user pool queue
users = Array["test1India@mailinator.com, test1US@mailinator.com",
"test2India@mailinator.com, test2US@mailinator.com",
"test3India@mailinator.com, test3US@mailinator.com",
"test4India@mailinator.com, test4US@mailinator.com"
]
users.each{|user| $users_queue.push(user)}

  • Iterate each scenario, Get credential from credential pool and run scenario on docker container.


# Iterate each test case, pop user credentials from pool queue
# And run scenario in docker container
count = 0
for testcase in $testcases.keys[0..$testcases.length] do
while $users_queue.empty?
process_container
end
count += 1
p "### #{count} ### RUNNING TEST #{testcase}"

users = $users_queue.pop
user = (!testcase.include? 'US') ? users.split(',')[0] : users.split(',')[1]
p "sudo docker run -d #{CONTAINER} "cucumber -p selenium #{testcase} app_env=#{APPENV}"
did = `sudo docker run -d #{CONTAINER} "cucumber -p selenium #{testcase} app_env=#{APPENV}"`
$dids_tc[did.strip] = testcase
$dids_users[did.strip] = users
end

while not $dids_tc.empty?
process_container
end

  • Retry failed scenarios for specified configuration for retry, Do log parsing and create reports, enqueue credential back in credential pool for waiting scenarios, and remove container.


# Process finished container for log and reports
# ReattemptRetry failed scenario for specified number of time
# Removed processedfinished container
def process_container
for did in $dids_tc.keys
state = `sudo docker inspect -f {{.State.Running}} #{did}`

if state.strip == false || state.strip == 'false'
testcase = $dids_tc[did]
users = $dids_users[did]
$dids_tc.delete(did)
$dids_users.delete(did)
$testcases[testcase] -= 1

output = `sudo docker logs #{did} | sed "s,x1B[[0-9;]*[a-zA-Z],,g"`
p testcase
output.split(/n/).each{|line| p line}

if (!output.include? "failed,") && (output.include? "passed)")
$logger.info("#######################################")
$logger.info(testcase)
$logger.info(output)
$testcases[testcase] = 0
$users_queue.push(users)
$PASSED += 1
else
if $testcases[testcase] > 0
p "### RETRYING ... #{RETRYCOUNT-$testcases[testcase]} TEST #{testcase}"

user = (!testcase.include? 'US') ? users.split(',')[0] : users.split(',')[1]
didtemp = `sudo docker run -d #{CONTAINER} "cucumber -p selenium #{testcase}"`
$dids_tc[didtemp.strip] = testcase
$dids_users[didtemp.strip] = users
else
$logger.info("#######################################")
$logger.info(testcase)
$logger.info(output)
#`docker cp #{did}:/opt/saas-shipping-ui/qa-automation/temp ../reports`
$users_queue.push(users)
$FAILED += 1
end
end

`sudo docker rm #{did}`
end
end
end

 
 

Source Code

  
  
Steps
  
1. Install Docker tool box

2. Get source from https://github.com/omarrohit20/parallel_test and change directory to parallel_test
  
3. Build docker image 
    docker build -t my_test .    
 
4. Run test with following command in parallel_test 
    ruby run_parallel.rb    
  
5. See logs on following location5. See logs on following location
  
    ../