How to capture Network log in Selenium 4 using CDP(Chrome DevTools Protocol)

package testing;
import java.util.Optional;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v131.network.Network;
import org.openqa.selenium.devtools.v131.network.model.Request;
import org.testng.annotations.Test;
public class Networklog extends BaseTest {
   @Test
   public void captureNetworkRequestLogs() {
      
    DevTools devTools = ((ChromeDriver)driver).getDevTools();
    devTools.createSession();
   
    devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
   
    devTools.addListener(Network.requestWillBeSent(), requestActivity ->{
   	   Request request = requestActivity.getRequest();
   	   System.out.println(request.getUrl());
   	         
    });
    driver.get("https://weatherstack.com/");
   }
}

Above Java code is using Selenium and Chrome DevTools Protocol (CDP) to capture network request details while visiting the specified webpage (https://weatherstack.com/).

In our Networklog  class, the output result will be a list of URLs corresponding to the network requests made by the page https://weatherstack.com/ during the test execution.

Here’s a breakdown of what happens:

1. DevTools Session: A session is created using devTools.createSession(), which connects the Selenium WebDriver (ChromeDriver) to the Chrome DevTools Protocol. This allows you to interact with the browser’s internal mechanisms.
2. Network Monitoring: The line devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty())) enables the capture of network events, like requests and responses.
3. Listener for Requests: The devTools.addListener(Network.requestWillBeSent(), …) sets up a listener for network requests that are sent during the page load. Every time a request is made, the listener is triggered.
4. Printing URLs: The listener extracts the URL of the network request (request.getUrl()) and prints it to the console using System.out.println(request.getUrl()).
5. Webpage Load: The line driver.get(“https://weatherstack.com/”); causes Selenium to load the page https://weatherstack.com/, and as the page loads, various network requests will be made (e.g., requests for images, scripts, stylesheets, API calls, etc.).

Output :- 

Output Description:

1. Request URLs: The console will output the URLs of all the network requests made by the browser when loading the https://weatherstack.com/ website.
   • You will see URLs corresponding to:
     • The main HTML page (e.g., https://weatherstack.com/).
     • API calls made to fetch data (e.g., JSON endpoints).
     • Static resources like CSS files, JavaScript files, images, or fonts requested by the page.
   • Each URL will be printed on a new line as System.out.println(request.getUrl()); is executed within the listener.
2. Timing: The requests will be logged in the order they are sent by the browser. The network listener captures the request before it is sent, so the URLs will appear in real time as they are initiated by the page load.

Capturing Chrome DevTools (CDT) results, such as network requests, using Selenium offers several benefits for web automation, debugging, and performance analysis. Here are some key advantages:

   Network Traffic Monitoring:–

   1. Detailed Network Insights: By capturing network requests, you can track all API calls, resources loaded (e.g., images, scripts), and responses. This is particularly useful for debugging issues related to resource loading or identifying failures in API requests.

   2. Latency and Timing Analysis: You can measure the time taken for different requests to complete, identify slow-loading resources, and optimize performance by analyzing the request and response times.

Monitoring External Services and API Interactions:-

API Testing: You can monitor interactions with APIs, checking whether correct requests are made and ensuring that the expected responses are returned. This is especially important for applications that rely heavily on external services.

Validating API Responses: You can also verify the data returned by APIs and ensure that it is correctly handled by the application (e.g., checking if the correct JSON data is parsed).