* Initial work on unittest framework, tested
* Finished reorganizing pjlib-test to use unit-test framework
* Fix big problem where performance improvement is not observed with the new framework (reason: because parallel flag is not set, doh!)
* Tidying up global vars in pjlib-test. Add test options: stop on error, skip essential tests, list tests
* Change the PJ_TEST_XX() signature to be more generic and does not force returning value
* Modifications to some existing tests to use unit-test test macros
* Updated VS projects with argparse.h and unittest.h
* Fix warnings on Windows
* Disable parallel unit-testing for ioqueue stress test on Windows because of errors when running on Windows virtual machine
* Non-parallel test case will now run exclusively (previously it did not wait the previous test to complete)
* Add pjlib-test/test_util.h for common utilities for parsing, configuring, and running unit test. This can be used by all test apps. pjlib-util-test has been ported to use this utilities
* Dirty hack to fix error message being displayed when user selected essential test because it does not exist in features test (in pjlib-test)
* Replace PJ_TEST_PARALLEL with PJ_TEST_EXCLUSIVE
* Ported pjnath-test to use unit-testing framework, with limited speed-up from 45m originally to 15m using 10 worker threads
* Large modifications in pjnath-test to speed-up test. It is fast now (4:30 minutes with 10 worker threads, from 45:42m originally)
* Ported pjmedia-test to use unit-test
* Porting of pjsip test to unit testing framework. Not much of speed improvements due to exclusive tests
* Modifications in pj_argparse API (changed get() to get_bool() and add automatic error reporting) hopefully make it easier to use
* Refactor tsx_uas_test() to allow parallel testing
* Parallelize tsx_uac_test()
* Further effort to parallize tests in pjsip-test. Discoverd major issue with unit-test logging (see unittest.md)
* Showing PJLIB config is optional with cmd line option
* Bug fixing failed pjsip tests when running in parallel mode
* Finished paralleizing all pjsip tests except one
* Continuing correcting errors
* Add test shuffle feature
* Modify CI workflows to use standard arguments: -w 3 --shuffle. These args are set in GitHub action variables
* Updated due to change in argparse API signature (swap arg order)
* Removed unittest.md (it was draft for the PR)
* Fix the use of GitHub repository vars
* Tidying up minor conflicting merge in VC projects and pjlib.h
* Attempt to fix assertion failure when tdata is being accessed after reference counter goes to zero in regc_test
* Fixing failure in transport_loop_test(), first in loop_resolve_error(), presumably because there are other loop transport around when the test is run. Solution are: 1) make the test exclusive for now, 2) fix the cleanup of loop transport in other tests. Then there is failure in transport_rt_test(), because it gets loop transport from other test that is being shutdown. Sneakily add pjsip_tpmgr_get_transport_count_by_type() that's useful for debugging.
* Attempt to fix unresolved winmm.lib API (e.g. timeKillEvent, timeGetTime) called from BaseClasses
* Split exclusive part of transport_loop_test
* CI mods: split steps, envs, shorten job names
* Fix failed tsx_basic_test
* Restore previously shortened job names
* Replace assertion with PJ_TEST_XX in resolver_test
* Add --stdout-buf and --stderr-buf options in test apps to control stdout/stderr buffering
* Fix missing sipp exe on Windows CI
* Fix wrong CI args on Mac. Use -j for faster make
* Replace pj_rand() with own rand in unittest because rand() yields different sequence on different platform even with the same srand, making it hard to reproduce test sequence
* Fix swig make error on Linux and runall.py error reading log file on Windows
* Attempt to fix test repeatability by 1) delete all calls to pj_srand() except in pj_test_suite_shuffle(), 2) unit test PRNG explicitly uses pj_uint32_t instead of int. Also disable windows python tests since it is unreliable
* Fixed inexistant function
* Relaxing the strictness of the test since sometimes it raises error
* Set regc_test() exclusive because it crashes sometimes, probably concurrency issue
* Modified thread counter to unsigned long (from pj_uint32_t) since the counter value is 2*1e9 and is overflow during diff calculation
* Fixed port double destruction in mips_test() and include benchmark tests in pj/config_site_test.h
* Use any port since sometimes test fails with address in use error
* Fix conflicted return value in udp_ioqueue_test() and let it immediately exit on error so that we can see correlated error log
* Restore sleep(0) in thread test since without it the test may occasionally fail on Linux
* Various attempt to fix fluke error in resolve_test.c: 1) servers use random port numbers, 2) increase delay waiting for various DNS timers, 3) reset global vars to zero because test may be repeated for IPv6
* More relaxed packet count tests in resolver_test
* Use any port instead of hardcoded one in udp ioqueue unregister_test since binding fails occasionally
* Rollback previous changes in resolver_test that relaxed packet count test
* Protect access to pool from worker thread with mutex in resolver_test
* Faster resolver_test time by reducing timeout
* Use high number port to make it less prone to bind error
* Remove hardcoded port number, replace with bind to any
* Fix SSL to continue decrypting data after renego completes
* Fixed race condition when registering SIP module in transport test
* Disable loading TLS cert in TURN sock test if SChannel is used
* Add (missing!) pjnath-test in Windows CI
* Fix syntax error in ci-win.yml and renamed lin->ubuntu in ci-linux
* Temporary workaround for MacOS rwmutex deadlock issue
* More jobs/tests in CI Mac, and changed names
* Fix silly typo in config_site_test.h
* Prettyfy CI job names, added more test steps
* Fix CI: ffmpeg lib path, faster git clone
* CI: simplify job names, add some audio checking
* CI: Disable SDL check because vid renderer is not available on CI machine
* Fixed missing ffmpeg shared lib when running pjmedia test
* CI Mac: install gnutls
* Lookup renderer in video codec test
* CI MacOS: attempt to fix failed OpenSSL test
* CI Mac: better test split to make duration more uniform across jobs
* Replace deprecated egrep with grep -E
---------
Co-authored-by: Nanang Izzuddin <nanang@teluu.com>
Co-authored-by: sauwming <ming@teluu.com>
0408536ef7