Chromedp Concurrency with Multiple Tabs

Understanding how to navigate chromedp concurrency with multiple tabs can significantly enhance your browser automation projects. This article guides you through the process, offering practical tips and clear understanding of this unique tech tool.

What is Chromedp?

For starters, chromedp is a programmer's ally. This powerful tool allows software developers to navigate the web in a programmatic way. Basically, it's a Go package that makes browser automation easier as it allows you to control Chrome (or any other Chrome-based browser) over the DevTools protocol. This allows you to simulate a user's operations on the browser, perform automated tasks, and much more.

func main() {
	startTime := time.Now()

	urls := []SSUrl{
		{URL: "https://keeplearning.dev/", Selector: `div.hero`, Prefix: "k"},
		{URL: "https://go.dev/", Selector: `img.Hero-gopherLadder`, Prefix: "go1"},
		{URL: "https://go.dev/", Selector: `img.Hero-gopherLadder`, Prefix: "go2"},
	}

	ctx, cancel := chromedp.NewContext(
		context.Background(),
		chromedp.WithDebugf(log.Printf),
	)

	defer cancel()

	var wg sync.WaitGroup
	const grMax = 5 // Number of goroutines to run concurrently

	ch := make(chan int, grMax)

	for _, url := range urls {
		// Increment the wait group counter for each URL

		wg.Add(1)
		ch <- 1

		// Execute tasks concurrently inside a goroutine

		go func(u SSUrl) {
			defer func() { wg.Done(); <-ch }()
			log.Printf("Capturing screenshot for URL: %s", u.URL)
			// same browser, second tab

			newCtx, _ := chromedp.NewContext(ctx)
			captureScreenshot(newCtx, u)
		}(url)
	}

	// Wait for all goroutines to finish

	wg.Wait()

	endTime := time.Now()
	scriptDuration := endTime.Sub(startTime)
	fmt.Println("All screenshots captured in => ", scriptDuration)
}

The critical part is this part;

newCtx, _ := chromedp.NewContext(ctx)

In this part, we are creating new tabs and we are using them with the goroutines.

To limit the number of simultaneously running goroutines we are using buffered channel. You can check this example for better understanding.