Speed up PHPUnit Code Coverage with PCOV

PCOV is a PHP driver used in PHPUnit for code coverage, similar to Xdebug or PHPDBG. It can be used to collect information about code coverage lines during testing.

Its main advantage over other code coverage analysis tools is its speed. It works faster and uses fewer resources.

About PCOV

PCOV was created by Joe Watkins solely focused on code coverage functionality without the additional features (profiling, tracing) found in other tools such as Xdebug and PHPDBG. This approach improves performance and speeds up testing.

Comparison of PCOV and Xdebug

PCOV and Xdebug are two different PHP tools used primarily for different tasks. However, both tools can be used to analyze code coverage when testing through PHPUnit.

With PCOV we can collect information about line coverage, while with Xdebug we have line, branch and path coverage. Read more about coverage types at the link.

PCOV has several advantages over Xdebug:

• Faster while running Code Coverage 2-4 times.
• Consumes fewer resources.

The disadvantages of PCOV over Xdebug include the following:

• Does not support path coverage and branch coverage.

Impact on PHP performance

PCOV has an impact on PHP performance when enabled. The performance impact is not such as noticeable as Xdebug, but you may encounter it when doing static analysis or running linter.
Activate it only for code coverage in the GitHub workflow with test runs or when running Code Coverage.

You can use the following command to activate it before running:
php -dpcov.enabled=1 ./vendor/bin/phpunit --coverage-text

UPD: PHPUnit errors in Code Coverage mode below version 9.5.20 when using @runInSeparateProcess

When using PHPUnit versions below 9.5.20 in Code Coverage mode with PCOV, if you run PCOV immediately before running Code Coverage, you may encounter errors in tests that use the @runInSeparateProcess annotation (which means run the test in a separate process):

PHP Fatal error:  Uncaught SebastianBergmann\CodeCoverage\NoCodeCoverageDriverAvailableException: No code coverage driver available in /var/www/html/vendor/phpunit/php-code-coverage/src/Driver/Selector.php:53
Stack trace:
#0 Standard input code(423): SebastianBergmann\CodeCoverage\Driver\Selector->forLineCoverage(Object(SebastianBergmann\CodeCoverage\Filter))
#1 Standard input code(1192): __phpunit_run_isolated_test()
#2 {main}
  thrown in /var/www/html/vendor/phpunit/php-code-coverage/src/Driver/Selector.php on line 53
Fatal error: Uncaught SebastianBergmann\CodeCoverage\NoCodeCoverageDriverAvailableException: No code coverage driver available in /var/www/html/vendor/phpunit/php-code-coverage/src/Driver/Selector.php:53
Stack trace:
#0 Standard input code(423): SebastianBergmann\CodeCoverage\Driver\Selector->forLineCoverage(Object(SebastianBergmann\CodeCoverage\Filter))
#1 Standard input code(1192): __phpunit_run_isolated_test()
#2 {main}
  thrown in /var/www/html/vendor/phpunit/php-code-coverage/src/Driver/Selector.php on line 53

This error tells us that the Code Coverage driver is missing and the phpunit/php-code-coverage package does not see it when running an isolated test.

This begs the question: Why doesn’t PHPUnit see the driver? Because we activate PCOV just before running Code Coverage.

The answer is that in PHPUnit before 9.5.20, the flags with which the PHP script was run were not passed to child processes spawned by PHPUnit Code Coverage. This means that the child processes are running with the PCOV driver disabled. For more details you can visit the issue on GitHub.

There are two solutions:

1. Update PHPUnit to 9.5.20 to include an updated version of the Code Coverage package with the fixed issue.
2. If you cannot upgrade to 9.5.20 for some reason, you need to activate PCOV in php.ini for the entire server in which you are running Code Coverage.

Compatibility

PCOV and Xdebug overload the same parts of the PHP engine, and as a result they are not compatible to work together.
If you need to use Xdebug for debugging or profiling, then you will need to disable PCOV and vice versa.

About the test bench

For the tests, I used the Laravel project BookStack. The test suite of the project has 950 tests, 4578 assertions and includes UnitTests and IntegrationTests. This set is more than enough to see the performance difference between PCOV and Xdebug.

All tests were run in GitHub Actions. BookStack already contains a customized isolated workflow for the tests. From test to test, I changed the coverage driver and PHP version in the shivammathur/setup-php@v2 GitHub Actions settings.

Each test was performed 3 times and the mean values were chosen for the article.

Test execution times are given in minutes.

The tests used PCOV version 1.0 on PHP 8.2 and 7.4

The commit on which the tests were performed 6e6f1133 in the BookStack repository.

Performance test on PHP 8.2

Let’s take a look at the difference in performance and resource consumption between PCOV and Xdebug on PHP 8.2.

PCOV

During the tests I got the following results

Time: 02:00, Memory: 184.50 MB

2 mitutes is pretty good for 950 tests и 4578 asserts.

Xdebug 3.2

Xdebug 3.2 was optimized quite well to reduce overhead. But Xdebug’s execution time is 2 times worse than PCOV’s. The difference in memory consumption is ~2mb, which is insignificant.

Time: 03:55, Memory: 186.50 MB

Performance test on PHP 7.4

PHP 7.4 is unsupported since November 28, 2022, but there are many projects that have not yet migrated to 8+ versions and for these projects the following tests may be useful.

PCOV

In tests on PHP 7.4 it showed the same execution time as PHP 8.2, but the memory consumption value is 2 times higher than PHP 8.2.

The difference in memory consumption in the tests for PHP 7.4 is due to optimizations made for PHP 8.* versions.

Time: 02:00, Memory: 347.00 MB

Xdebug 3.1

Xdebug 3.1 is slower by 2.5 times compared to PCOV on PHP 7.4. The difference in memory is again insignificant ~6mb.

Time: 04:56, Memory: 353.00 MB

Xdebug 2.9

Of course, I don’t recommend you to use Xdebug version 2, because it is very slow. But let’s see what the results are in case you still use this version.

Tests have shown that Xdebug 2.9 is 4 times slower than PCOV on PHP 7.4.

Time: 07:59, Memory: 355.00 MB

Test results

Code Coverage test summary table:

As you can see:

• PCOV is faster than Xdebug 3.2 (PHP 8.2) by 197%
• PCOV is faster than Xdebug 3.1 (PHP 7.4) by 247%
• PCOV is faster than Xdebug 2.9 (PHP 7.4) by 400%
• Xdebug 3.2 (PHP 8.2) is 25% faster than Xdebug 3.1 (PHP 7.4).
• Xdebug 3.1 is 60% faster than Xdebug 2.9 (PHP 7.4).

Conclusion

PCOV beats Xdebug in speed of execution of tests with code coverage by 2-4 times. This may not be significant if you use unit tests that take seconds to execute. But what if your project has also integration tests with a set of 5000 tests, 40000 assertions, which takes from 5 minutes to complete?

In this case, the performance difference of 2-4 times between PCOV and Xdebug can be significant and it can affect the speed of CI/CD and the frequency of running tests by the programmer and therefore the speed of development new features due to fixing bugs at a late stage.

It is also important to remember that PCOV, like Xdebug, can affect application performance. Activate it only for code coverage in the GitHub test workflow or just before executing tests.