Selenium - Execute POST request asynchronously

One might feel limited in Selenium being unable to verify certain things that interacts through XHR Ajax Requests especially when they are http POST calls. This could be applicable when you are trying to crawl a particular website and you are not able to get a certain datapoint because it is hiding behind a XHR Post in the page.

There are many ways to solve this problem, I am showing you two using Ruby Selenium Webdriver, however the concept is applicable across any language.

1. Using Local Storage

Assuming, there is an endpoint called postEndPoint and we want to get its value, we can use browser’s localstorage object.

fetch("http://testenviron.com/postEndPoint", {
 method: "POST",
 headers: {blah: blah},
 body: { a: 1 }
}).then(data => {
 localStorage.setItem("myKey", JSON.stringify(data))
});

We can load the above script in a variable called script and use the executeScript method of Selenium like shown below.

browser.execute_script(script)
sleep(1)
browser.local_storage.fetch("myKey")

If you observe here, the obvious drawback is the use of sleep statement which blocks the main thread. Even worse is, when you do not tell for sure how long the call is going to take, you might end up increasing the sleep more! This is not clean enough, but it does the job if you know that the call is going to return fast and is “predictable”

2. execute_async_script

If you want to do things asynchronously or rather “cleanly”, we can use the execute_async_script feature which does exactly what it does with a callback. The callback will not be executed until the execution is complete.

script = "
  var callback = arguments[arguments.length - 1];
  setTimeout(function(){
    callback(['hello world'])
  }, 3000);
"
browser.execute_async_script(script)
=> ["hello world"]

This is the cleanest way because the main thread will be listening and waiting for the callback automatically. No need to sleep or do any kind of hack here!

Francium Tech is a technology company laser focused on delivering top quality software of scale at extreme speeds. Numbers and Size of the data don’t scare us. If you have any requirements or want a free health check of your systems or architecture, feel free to shoot an email to contact@francium.tech, we will get in touch with you!