When I started using selenium and UI testing tools, my first experience with headless browsers was with docker, by using selenium/standalone-chrome image, which is very easy to use, since gives a running selenium with all deps. After some time, I was working with Azure Devops and running my tests without docker, directly with webdriver (chromedriver and safaridriver, in my case). To avoid issues with the agent, I was using XVFB to generate a new window and run the tests there. It’s a pretty simple code:

*** Settings ***
Documentation     This example demonstrates how to use current library
Library           Selenium2Library
Library           XvfbRobot

*** Test Cases ***
Create Headless Browser
    Start Virtual Display    1920    1080
    Open Browser   http://google.com

But, when I was trying to do the same on macos, I got some trouble to install xvfb by brew. So reading more careful the documentation, I saw that some drivers(Chrome and Firefox) provide this feature native, just select then when opening a browser:

*** Test Cases ***
Open Headless Chrome Browser
    Open Browser   http://google.com    browser=headlesschrome

Open Headless Firefox Browser
    Open Browser   http://firefox.com    browser=headlessfirefox

You can access more info on SeleniumLibrary docs.