Design Specification for ./mach manifest skip-fails =================================================== The first of the mach manifest subcommands is skip-fails. This command can be used to automatically edit manifests to skip tests that are failing as well as file the corresponding bugs for the failures. This is particularly useful when “greening up” a new platform. The user documentation for skip-fails is here: https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html#using-mach-manifest-skip-fails The skip-fails command knows how to manage failures for TOML, WPT and REFTEST manifests. Additional developer design notes are included here. Running skip-fails for TOML --------------------------- The user documentation for TOML manifests is: https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html https://firefox-source-docs.mozilla.org/testing/tests-for-new-config/index.html#run-skip-fails Usage ----- $ ./mach manifest skip-fails -h usage: mach [global arguments] manifest skip-fails [command arguments] Global Arguments: -v, --verbose Print verbose output. --quiet Don't print as much output. -l, --log-file FILENAME Filename to write log data to. --log-interval Prefix log line with interval from last message rather than relative time. Note that this is NOT execution time if there are parallel operations. --no-interactive Automatically selects the default option on any interactive prompts. If the output is not a terminal, then --no-interactive is assumed. --log-no-times Do not prefix log lines with times. By default, mach will prefix each output line with the time since command start. -h, --help Show this help message. --debug-command Start a Python debugger when command is dispatched. --profile-command Capture a Python profile of the mach process as command is dispatched. --settings FILENAME Path to settings file. Sub Command Arguments: try_url Treeherder URL for try (please use quotes) -b, --bugzilla BUGZILLA Bugzilla instance [disable] -C, --clear-cache [CLEAR_CACHE] clear cache REVISION (or all) -c, --carryover Set carryover mode (only skip failures for platform matches) -d, --dry-run Determine manifest changes, but do not write them -F, --use-failures USE_FAILURES Use failures from file -f, --save-failures SAVE_FAILURES Save failures to file -I, --implicit-vars Use implicit variables in reftest manifests -i, --task-id TASK_ID Task id to write a condition for instead of all tasks from the push -k, --known-intermittents Set known intermittents mode (only skip failures known intermittents) -M, --max-failures MAX_FAILURES Maximum number of failures to skip (-1 == no limit) -m, --meta-bug-id META_BUG_ID Meta Bug id -n, --new-version NEW_VERSION New version to use for annotations -N, --new-failures Set new failures mode (only add conditions for new failures) -r, --failure-ratio FAILURE_RATIO Ratio of test failures/total to skip [0.4] -R, --replace-tbd Replace Bug TBD in manifests by filing new bugs -s, --turbo Skip all secondary failures -T, --use-tasks USE_TASKS Use tasks from file -t, --save-tasks SAVE_TASKS Save tasks to file -u, --user-agent USER_AGENT User-Agent to use for mozci if queries are forbidden from treeherder -v, --verbose Verbose mode Design ------ 1. Classify the failure based on the following: a. If less than 3 runs, classify "unknown" b. If zero failures, classify "success" c. If the failure ratio of failed runs / total runs is less than 40% classify "intermittent" d. Else edit the manifest to skip this failure 2. Manifest variable information a. Variables used in Python are set: https://searchfox.org/firefox-main/source/testing/mozbase/mozinfo/mozinfo/platforminfo.py NOTE: that the task metadata runtimes are turned in to test variants by looking up the "mozinfo" field that corresponds to the "runtime" key in taskcluster/test_configs/variants.yml NOTE: special excpetions are these runtimes are mapped to these variants: - no-fission -> !fission - 1proc -> !e10s NOTE: multiple variants are stored in the field test_variant concatenated by '+', for example: "socketprocess_networking+!fission" b. Variables used in JavaScript are set: https://searchfox.org/firefox-main/source/layout/tools/reftest/manifest.sys.mjs c. Use arch instead of processor https://searchfox.org/firefox-main/source/testing/mozbase/mozinfo/mozinfo/mozinfo.py#170 aarch64 arm7 x86 x86_64 d. Build types An an optimized build "opt" implies none of these build types: "debug", "ccov", "asan", "tsan". A build type "!debug" implies "opt" e. A "platform match" for the purpose of bug carryover means - "os" is not specified in the existing condition or "os" is the same, and - The existing build type is not present and the "os" explictly the same, or - The existing build type is present: + "opt" (or "!debug") in both existing skip-if and proposed skip-if + "!opt" or one of ["debug", "asan", "tsan", "ccov" ] in both (covers the case of "debug" in both) EXCEPT does not match if one is "debug" and the other is in ["asan", "tsan", "ccov"] - Else not a match f. If the argument `-n, --new-version NEW_VERSION` is used then the os_version used in any skip-if expression will be NEW_VERSION -- regardless of the os, arch or another actual platform information. This is intended to workaround a bug where task metadata would report "macosx 10.15", yet the worker was actually "macosx 14.70". By setting `--new-version 14.70` then any skip-if expression will be created with `os_version == 14.70` which would then match what mozinfo says on the next try run. 3. Bugzilla interaction a. The bugzilla instance will be set to (in increasing precedence): i. Default: "bugzilla.allizom.org" ii. From the environment variable BUGZILLA iii. From the command line arg --bugzilla b. NOTE: if the BUGZILLA env var or command line arg is "disable" then Bugzilla interaction will be disabled. c. Setting --dry-run prevents editing manifests, but will not disable reading Bugzilla. Nevertheless no new bugs will be created nor comments added. Note existing bugs may be mentioned in an info message as appropriate. d. If --meta-bug-id is NOT set no bugs will be filed. Any created manifest bugs will require referencing a meta-bug-id. e. If there is a skip line for this manifest and the platform matches (see above) then add an info message about carryover do NOT edit any existing bug nor create a new bug If there is an existing manifest comment, then carry it over to the new skip line Else do NOT add a new comment (as a comment did not exist before) Else (no skip lines match the platform) Add a new skip-if line and reference the bug for this manifest + meta-bug-id (create a new bug if needed) f. Special case for carryover when a [default] section exists and contains a skip-if with a platform match AND our current failure matches that skip-if line for a real manifest THEN carry over the comment from [default] to the new skip-if line. 4. Modes for skip-fails a. Normal mode Tests with failures found based on the above heuristics are disabled with skip-if conditions in the appropriate manifests and bugs filed (or updated). b. Carry-over mode Only consider adding skip-if conditions which match the platform (see above) of previous conditions. Does not perform any bugzilla changes. Requires meta-bug-id ( future actions will be cached in {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) c. Known intermittents mode Only consider adding skip-if conditions which have known intermittent bugs (and are not carryover bugs). form (see above) of previous conditions. Does not perform any bugzilla changes. Requires meta-bug-id ( future actions will be cached in {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) For each failure, if the job_id can be determined then bug_suggestions will be retrieved from https://treeherder.mozilla.org/api/project/{repo}/jobs/{job_id}/bug_suggestions/ each object with "search" field of ^PROCESS-CRASH or ^TEST-UNEXPECTED will be considered if one of the bugs in bugs.open_recent..summary ends with 'single tracking bug$' then bugs.open_recent..id has the bug id d. New failures mode Will only edit manifest skip-if conditions for new failures (i.e. not carryover nor known intermittents). Will record bug as "TBD". Requires meta-bug-id ( future actions will be cached in {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) e. Replace TBD mode Will only edit manifest skip-if conditions for new failures by filing new bugs and replacing TBD with actual bug number. Requires meta-bug-id ( new bug actions will be read from {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) 5. Cache for skip-fails At the top of the source tree is a cache directory for skip-fails: ".skip_fails_cache". This directory will be created if it is not present. When running skip-fails if --use-tasks has not been set then the cache directory will be searched for a tasks file corresponding the revision (i.e. ".skip_fails_cache/REVISION/tasks.json"). If that file is present then it is used. Similarly for --use-failures. If --save-tasks has not been set then, by default, the tasks will be saved in the cache directory. Similarly for --save-failures. Any specific cache directory can be cleared with --clear-cache REVISION. All cache directories can be deleted with --clear-cache (no argument). Any cache directories which are over 45 days old will be automatically cleared. As design changes for skip-fails / TOML, notes will be added here. Running skip-fails for WPT --------------------------- The user documentation for WPT manifests is: https://firefox-source-docs.mozilla.org/web-platform/index.html 1. Classify the failure based on the following: a. If less than 3 runs, classify "unknown" b. If zero failures, classify "success" c. If the ratio of failed runs / total runs is less than 40% classify "intermittent" d. Else edit the manifest to skip this failure As design changes for skip-fails / WPT, notes will be added here. Running skip-fails for REFTEST ------------------------------ The user documentation for REFTEST manifests is: https://firefox-source-docs.mozilla.org/layout/Reftest.html Current Design notes for skip-fails / REFTEST: As skip-fails for REFTEST finds a failure it will take the following actions: 1. Classify the failure based on the following: a. If less than 3 runs, classify "unknown" b. If zero failures, classify "success" c. If the ratio of failed runs / total runs is less than 40% classify "low frequency intermittent" (INTERMITTENT) c. If the ratio of failed runs / total runs is less than 80% classify "high frequency intermittent" (DISABLE_INTERMITTENT) d. If the ratio is <= 100%, classify "failure" (DISABLE_FAILURE) 2. Propose a new range (differences, pixels) a. For "low frequency intermittent" errors do nothing b. For "high frequency intermittent" set the range 0-Y (i.e. always start at zero to avoid UNEXPECTED PASS) c. For "failure" set the range X-Y (i.e. the range as reported in reftest_errorsummary.log). If the failure is UNEXPECTED PASS then set the range to 0-Y. d. NOTE: The maximum for differences and pixels will be increased by 5% over the reported values in reftest_errorsummary.log e. NOTE: If for whatever reason the range is calculated to be 0-0 then remove the fuzzy-if entirely 3. Merge with an existing fuzzy-if for this OS platform if the other criteria differ in only one dimension. Current dimensions include: OS: gtkWidget, winWidget, cocoaWidget, Android build_type: optimized, isDebugBuild, isCoverageBuild, AddressSanitizer, ThreadSanitizer is64Bit fission useDrawSnapshot swgl 4. Based on a command line switch choose do use implicit variables (or not). a. The following variables can implicit values: is64Bit: true fission: true useDrawSnapshot: false swgl: false If the value of an implicit variable matches it's default then it may be elided from the conditional expression. If the value of an implicit variable is the oppposite of it's default value then it may be specified in the condition, e.g. !is64Bit When merging an existing condition with a new failure condition where the one dimension of difference is an implicit variable then the condition must include the disjunction of the variable, e.g. (is64Bit&&!is64Bit) That is essential so that the reftest sandbox does not add the implicit variable and cover only one of the two possible values. b. When implicit variables are not used then the following heuristics apply: The variable is64Bit will be elided for all platforms except for winWidget. The variable useDrawSnapshot will be elided for all platforms except gtkWidget. 5. Position of fuzzy-if conditions In reftest manifest the rightmost applicable condition is used. a. First sorting criterion is based on OS in this order: OSES = ["Android", "cocoaWidget", "appleSilicon", "gtkWidget", "winWidget"] This is because appleSilicon is a narrower specification than cocoaWidget and would be ignored if not to the right of cocoaWidget. b. The second sorting criterion is the number of dimensions that participate in the expression (including implicits) from least dimensions to greatest dimensions. The last fuzzy-if will be placed on the line immediately to the left of the "type" (e.g. == or !=) (or to the left of an optional "HTTP" expression, if present). Non fuzzy-if statements will appear before the fuzzy-if statements. In this way the more generic conditions will only be superceded by very specific conditions. Design Specification for ./mach manifest clean-skip-fails ========================================================= The purpose / rationale for this command is to-be-documented. description="Update manifests to remove skip-if conditions for a specific platform. Only works for TOML manifests.", The user documentation is: https://firefox-source-docs.mozilla.org/... Usage ----- Sub Command Arguments: manifest_search_path Path to the folder containing the manifests to update, or the path to a single manifest -o, --os OS_NAME OS to remove (linux, mac, win) -s, --os_version OS_VERSION Version of the OS to remove (eg: 18.04 for linux) -p, --processor PROCESSOR Type of processor architecture to remove (eg: x86) Design Specification for ./mach manifest high-freq-skip-fails ========================================================= The purpose / rationale for this command is to-be-documented. description="Update manifests to skip failing tests", The user documentation is: https://firefox-source-docs.mozilla.org/... Usage ----- Sub Command Arguments: -f, --failures FAILURES Minimum number of failures for the bug to be skipped -d, --days DAYS Number of days to look for failures since now