运行测试用例
测试用例准备好后,运行hrun
执行测试用例。
注意:hrun
是httprunner run
的缩写,他们功能一样。
hrun = httprunner run
运行用例的各种方式
(1)运行指定路径的用例
hrun path/to/testcase1
(2)指定多个测试用例路径执行
hrun path/to/testcase1 path/to/testcase2
(3)运行测试项目中所有测试用例 指定测试项目的绝对路径
hrun path/to/testcase_folder/
运行YAML/JSON格式的测试用例
如果测试用例格式是YAML或JSON,hrun
先将YAML/JSON转为pytest(python)格式,然后再运行pytest。
相当于:
hrun = make + pytest
对于大多数用例,生成的pytest文件和YAML/JSON保存在相同目录下。生成的pytest用例名增加_test
后缀,且.yml/yaml/.json
替换为.py
。
/path/to/example.yml => /path/to/example_test.py
然而,如果测试用例目录名或用例文件名包含. - 空格
这些字符,这些字符将被_
替换,以避免python类中用例引用时的语法错误。
而且,以数字开头的目录和文件名,将添加T
前缀,因为python模块和类名不能以数字开头。
path 1/a.b-2/3.yml => path_1/a_b_2/T3_test.py
运行pytest用例
当想运行pytest格式用例 或 YAML/JSON转换后的pytest用例,hrun
和pytest
都可以使用。
需要记住的是,hrun
只包装了pytest
,所以pytest
的功能都可以在hrun
中使用。
$ hrun -h
usage: hrun [options] [file_or_dir] [file_or_dir] [...]
positional arguments:
file_or_dir
general:
-k EXPRESSION 仅运行与给定字符串表达式匹配的测试。表达式是python可识别的表达式,其中所有名称均为与测试文件名称及其父类进行子字符串匹配。
例如:`-k 'test_method or test_other'` 匹配所有测试方法名和类名包含'test_method' 或 'test_other'的用例。
`-k 'not test_method'` 匹配名称中不包含'test_method'的类。
`-k 'not test_method and not test_other'` 消除匹配项。
另外,还运行,在其他的关键字匹配中,类和方法名成包含的,以及直接为其分配名称的函数。
匹配不区分大小写。
-m MARKEXPR 只运行匹配给定表达式匹配的用例。 例如: -m 'mark1 and not mark2'.
--markers 显示标记(内置标记,插件标记和只运行一次标记)
-x, --exitfirst 遇到error或fail test立即退出.
--maxfail=num 在第一个失败用例或错误后退出。
--strict-markers, --strict
未在配置文件中"标记"部分注册的标记会引发错误。
-c file 从'file'加载配置,而不是加载隐藏的配置文件。
--continue-on-collection-errors
即使发生错误,也强制继续执行测试。
--rootdir=ROOTDIR 定义测试项目的root目录. 可以是相对路径:'root_dir', './root_dir', 'root_dir/another_dir/'; 绝对路径:
'/home/user/root_dir'; 包含变量的路径: '$HOME/root_dir'.
--fixtures, --funcargs
显示可用的fixtures, 按fixture名称排序 (以`_`开头的fixture,只有添加`-v`时才显示)
--fixtures-per-test 每次测试都显示fixtures
--import-mode={prepend,append}
当import测试模块时, prepend/append到系统环境变量中,默认前置到系统环境变量中。
--pdb 当出现error或强制断开时,启动交互式python调试器.
--pdbcls=modulename:classname
针对错误启动自定义交互式python调试器. 例如: --pdbcls=IPython.terminal.debugger:TerminalPdb
--trace 每次运行测试时立即跳出
--capture=method 每次测试都捕获方法: fd|sys|no|tee-sys的其中之一。
-s --capture=no的快捷模式.
--runxfail 报告xfail标记的测试结果,就像没标记过一样。
--lf, --last-failed 仅重新运行最后一次运行失败的测试(如果没有失败的则全部重新运行)
--ff, --failed-first 运行所有测试,但先运行最后的失败用例。这可能会导致setup/teardown的重复运行。
--nf, --new-first 首先运行新的测试用例,然后其余测试用例按照最后一次修改时间顺序执行。
--cache-show=[CACHESHOW]
显示缓存内容, 不执行搜集或测试. 可选参数: glob (default: '*').
--cache-clear 测试运行前清空所有缓存.
--lfnf={all,none}, --last-failed-no-failures={all,none}
运行没有失败过的用例.
--sw, --stepwise 测试失败时退出,且下次运行从最后失败的用例开始运行
--stepwise-skip 忽略第一个失败的测试,下一个失败的用例出现时停止
--allure-severities=SEVERITIES_SET
以逗号分割的优先级名称列表. 只运行指定优先级的用例. 可选择的优先级为: blocker, critical, normal, minor, trivial.
--allure-epics=EPICS_SET
以逗号分割的epic列表. 运行具有至少一个指定标签的测试。
--allure-features=FEATURES_SET
以逗号分割的feature列表. 运行具有至少一个指定标签的测试。
--allure-stories=STORIES_SET
以逗号分割的story列表. 运行具有至少一个指定story的测试.
--allure-link-pattern=LINK_TYPE:LINK_PATTERN
执行命令时指定短链接, 比如Bug地址. 格式为python str.format()格式,例如:`pytest test.py ----allure-link-pattern=issue:http//www.mytesttracker.com/issue/{}`.
reporting:
--durations=N 显示最慢的N个测试用例和setup (N=0 for all).
-v, --verbose 增加报告信息.
-q, --quiet 减少报告信息.
--verbosity=VERBOSE 设置详细程度,默认是0。
-r chars 显示字符指定的测试摘要信息: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed
(a)ll except passed (p/P), or (A)ll. 默认启用(w)arnings (see --disable-warnings), 'N' 用于重置列表
(默认: 'fE').
--disable-warnings, --disable-pytest-warnings
禁用警告摘要
-l, --showlocals 在回溯中显示局部变量(默认禁止).
--tb=style 追溯打印模式 (auto/long/short/line/native/no).
--show-capture={no,stdout,stderr,log,all}
控制在失败的测试中如何显示捕获的stdout/stderr/log. 默认'all'.
--full-trace 不删减任何信息(默认是删减的).
--color=color 彩色终端输出 (yes/no/auto).
--pastebin=mode 发送失败|all信息到bpaste.net服务
--junit-xml=path 在指定路径生成junit-xml报告
--junit-prefix=str 在junit-xml输出中添加str到类名前面
--result-log=path 可读结果日志的已弃用路径
--html=path 指定路径生成html报告
--self-contained-html
创建一个包含所有必需的样式,脚本和图像的自包含html文件 - 这意味着该报告可能不会呈现或具备CSP限制的功能 (见: https://developer.mozilla.org/docs/Web/Security/CSP)
--css=path 将给定的CSS文件内容附加到报表样式文件中。
collection:
--collect-only, --co 仅收集测试,不执行测试。
--pyargs 尝试将所有参数解释为python软件包。
--ignore=path 收集期间忽略指定路径(允许多个路径).
--ignore-glob=path 收集期间忽略路径 (允许多个路径).
--deselect=nodeid_prefix
在收集期间取消选择项(通过节点id前缀)(允许多个)
--confcutdir=dir 仅加载指定目录(conftest.py的相对路径)
--noconftest 不加载任何conftest.py files.
--keep-duplicates 保持重复测试
--collect-in-virtualenv
不要忽略本地virtualenv目录中的测试
--doctest-modules 在所有.py模块中运行doctests
--doctest-report={none,cdiff,ndiff,udiff,only_first_failure}
为doctest失败时的差异选择另一种输出格式
--doctest-glob=pat doctests文件匹配模式,默认值:test * .txt
--doctest-ignore-import-errors
忽略 doctest ImportErrors
--doctest-continue-on-failure
对于给定的doctest,在第一次失败后继续运行
测试会话调试和配置:
--basetemp=dir 此测试运行的基本临时目录。(警告:如果存在此目录,则将其删除)
-V, --version 显示pytest版本和有关插件的信息。
-h, --help 显示帮助消息和配置信息
-p name 尽早加载给定的插件模块名称或入口点(允许多个). 为避免加载插件, 使用`no:` prefix, 例如. `no:doctest`.
--trace-config 跟踪conftest.py文件的注意事项。
--debug 将内部跟踪调试信息存储在“ pytestdebug.log”中。
-o OVERRIDE_INI, --override-ini=OVERRIDE_INI
以“option = value”样式覆盖ini选项,例如 `-o xfail_strict = True -o cache_dir = cache`
--assert=MODE 控制断言调试工具。 'plain'不执行任何断言调试。 “ rewrite”(默认)在导入时重写测试模块中的assert语句,以提供assert表达式信息。
--setup-only 仅安装fixtures, 不执行测试.
--setup-show 执行测试时显示fixtures的setup.
--setup-plan 不执行,只显示将要执行的fixture和测试用例
pytest-warnings:
-W PYTHONWARNINGS, --pythonwarnings=PYTHONWARNINGS
set which warnings to report, see -W option of python itself.
logging:
--no-print-logs disable printing caught logs on failed tests.
--log-level=LEVEL level of messages to catch/display. Not set by default, so it depends on the root/parent log handler's effective level, where it is
"WARNING" by default.
--log-format=LOG_FORMAT
log format as used by the logging module.
--log-date-format=LOG_DATE_FORMAT
log date format as used by the logging module.
--log-cli-level=LOG_CLI_LEVEL
cli logging level.
--log-cli-format=LOG_CLI_FORMAT
log format as used by the logging module.
--log-cli-date-format=LOG_CLI_DATE_FORMAT
log date format as used by the logging module.
--log-file=LOG_FILE path to a file when logging will be written to.
--log-file-level=LOG_FILE_LEVEL
log file logging level.
--log-file-format=LOG_FILE_FORMAT
log format as used by the logging module.
--log-file-date-format=LOG_FILE_DATE_FORMAT
log date format as used by the logging module.
--log-auto-indent=LOG_AUTO_INDENT
Auto-indent multiline messages passed to the logging module. Accepts true|on, false|off or an integer.
reporting:
--alluredir=DIR Generate Allure report in the specified directory (may not exist)
--clean-alluredir Clean alluredir folder if it exists
--allure-no-capture Do not attach pytest captured logging/stdout/stderr to report
custom options:
--metadata=key value additional metadata.
--metadata-from-json=METADATA_FROM_JSON
additional metadata from a json string.
[pytest] ini-options in the first pytest.ini|tox.ini|setup.cfg file found:
markers (linelist): markers for test functions
empty_parameter_set_mark (string):
default marker for empty parametersets
norecursedirs (args): directory patterns to avoid for recursion
testpaths (args): directories to search for tests when no files or directories are given in the command line.
usefixtures (args): list of default fixtures to be used with this project
python_files (args): glob-style file patterns for Python test module discovery
python_classes (args):
prefixes or glob names for Python test class discovery
python_functions (args):
prefixes or glob names for Python test function and method discovery
disable_test_id_escaping_and_forfeit_all_rights_to_community_support (bool):
disable string escape non-ascii characters, might cause unwanted side effects(use at your own risk)
console_output_style (string):
console output: "classic", or with additional progress information ("progress" (percentage) | "count").
xfail_strict (bool): default for the strict parameter of xfail markers when not given explicitly (default: False)
enable_assertion_pass_hook (bool):
Enables the pytest_assertion_pass hook.Make sure to delete any previously generated pyc cache files.
junit_suite_name (string):
Test suite name for JUnit report
junit_logging (string):
Write captured log messages to JUnit report: one of no|log|system-out|system-err|out-err|all
junit_log_passing_tests (bool):
Capture log information for passing tests to JUnit report:
junit_duration_report (string):
Duration time to report: one of total|call
junit_family (string):
Emit XML for schema: one of legacy|xunit1|xunit2
doctest_optionflags (args):
option flags for doctests
doctest_encoding (string):
encoding used for doctest files
cache_dir (string): cache directory path.
filterwarnings (linelist):
Each line specifies a pattern for warnings.filterwarnings. Processed after -W/--pythonwarnings.
log_print (bool): default value for --no-print-logs
log_level (string): default value for --log-level
log_format (string): default value for --log-format
log_date_format (string):
default value for --log-date-format
log_cli (bool): enable log display during test run (also known as "live logging").
log_cli_level (string):
default value for --log-cli-level
log_cli_format (string):
default value for --log-cli-format
log_cli_date_format (string):
default value for --log-cli-date-format
log_file (string): default value for --log-file
log_file_level (string):
default value for --log-file-level
log_file_format (string):
default value for --log-file-format
log_file_date_format (string):
default value for --log-file-date-format
log_auto_indent (string):
default value for --log-auto-indent
faulthandler_timeout (string):
Dump the traceback of all threads if a test takes more than TIMEOUT seconds to finish. Not available on Windows.
addopts (args): extra command line options
minversion (string): minimally required pytest version
render_collapsed (bool):
Open the report with all rows collapsed. Useful for very large reports
environment variables:
PYTEST_ADDOPTS extra command line options
PYTEST_PLUGINS comma-separated plugins to load during startup
PYTEST_DISABLE_PLUGIN_AUTOLOAD set to disable plugin auto-loading
PYTEST_DEBUG set to enable debug tracing of pytest's internals
to see available markers type: pytest --markers
to see available fixtures type: pytest --fixtures
(shown according to specified file_or_dir or current dir if not specified; fixtures with leading '_' are only shown with the '-v' option
执行日志
$ hrun examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:39:41.041 | INFO | httprunner.make:make_testcase:317 - start to make testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:39:41.042 | INFO | httprunner.make:make_testcase:390 - generated testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
2020-06-17 15:39:41.042 | INFO | httprunner.make:format_pytest_with_black:154 - format pytest cases with black ...
reformatted /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
2020-06-17 15:39:41.315 | INFO | httprunner.cli:main_run:56 - start to run tests with pytest. HttpRunner version: 3.0.13
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner
plugins: metadata-1.9.0, allure-pytest-2.8.16, html-2.1.1
collected 1 item
examples/postman_echo/request_methods/request_with_functions_test.py . [100%]
======================================================================= 1 passed in 2.98s =======================================================================
如果你想看到request和response、提取和断言的详细信息,可以增加-s
(--capture=no
的缩写)
$ hrun -s examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:42:54.369 | INFO | httprunner.make:make_testcase:317 - start to make testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:42:54.369 | INFO | httprunner.make:make_testcase:390 - generated testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
2020-06-17 15:42:54.370 | INFO | httprunner.make:format_pytest_with_black:154 - format pytest cases with black ...
reformatted /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
2020-06-17 15:42:54.699 | INFO | httprunner.cli:main_run:56 - start to run tests with pytest. HttpRunner version: 3.0.13
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner
plugins: metadata-1.9.0, allure-pytest-2.8.16, html-2.1.1
collected 1 item
examples/postman_echo/request_methods/request_with_functions_test.py 2020-06-17 15:42:55.017 | INFO | httprunner.runner:test_start:435 - Start to run testcase: request methods testcase with functions, TestCase ID: cc404c49-000f-485c-b4c1-ac3367a053fe
2020-06-17 15:42:55.018 | INFO | httprunner.runner:__run_step:278 - run step begin: get with params >>>>>>
2020-06-17 15:42:56.326 | DEBUG | httprunner.client:log_print:40 -
================== request details ==================
method : GET
url : https://postman-echo.com/get?foo1=bar11&foo2=bar21&sum_v=3
headers : {
"User-Agent": "HttpRunner/3.0.13",
"Accept-Encoding": "gzip, deflate",
"Accept": "*/*",
"Connection": "keep-alive",
"HRUN-Request-ID": "HRUN-cc404c49-000f-485c-b4c1-ac3367a053fe-775018",
"Content-Length": "2",
"Content-Type": "application/json"
}
cookies : {}
body : {}
2020-06-17 15:42:56.327 | DEBUG | httprunner.client:log_print:40 -
================== response details ==================
status_code : 200
headers : {
"Date": "Wed, 17 Jun 2020 07:42:56 GMT",
"Content-Type": "application/json; charset=utf-8",
"Content-Length": "477",
"Connection": "keep-alive",
"ETag": "W/\"1dd-2JtBYPcnh8D6fqLz8KFn16Oq1R0\"",
"Vary": "Accept-Encoding",
"set-cookie": "sails.sid=s%3A6J_EtUk3nkL_C2xtx-NtAXrlA5wPxEgk.gIO2yBbtvGWIIgQ%2F2mZhMkU669G3F60cvLAPWbwyoGM; Path=/; HttpOnly"
}
cookies : {
"sails.sid": "s%3A6J_EtUk3nkL_C2xtx-NtAXrlA5wPxEgk.gIO2yBbtvGWIIgQ%2F2mZhMkU669G3F60cvLAPWbwyoGM"
}
encoding : utf-8
content_type : application/json; charset=utf-8
body : {
"args": {
"foo1": "bar11",
"foo2": "bar21",
"sum_v": "3"
},
"headers": {
"x-forwarded-proto": "https",
"x-forwarded-port": "443",
"host": "postman-echo.com",
"x-amzn-trace-id": "Root=1-5ee9c980-d8e98cc72a26ef24f5819ce3",
"content-length": "2",
"user-agent": "HttpRunner/3.0.13",
"accept-encoding": "gzip, deflate",
"accept": "*/*",
"hrun-request-id": "HRUN-cc404c49-000f-485c-b4c1-ac3367a053fe-775018",
"content-type": "application/json"
},
"url": "https://postman-echo.com/get?foo1=bar11&foo2=bar21&sum_v=3"
}
2020-06-17 15:42:56.328 | INFO | httprunner.client:request:203 - status_code: 200, response_time(ms): 1307.33 ms, response_length: 477 bytes
2020-06-17 15:42:56.328 | INFO | httprunner.response:extract:152 - extract mapping: {'foo3': 'bar21'}
2020-06-17 15:42:56.328 | INFO | httprunner.response:validate:209 - assert status_code equal 200(int) ==> pass
2020-06-17 15:42:56.329 | INFO | httprunner.response:validate:209 - assert body.args.foo1 equal bar11(str) ==> pass
2020-06-17 15:42:56.329 | INFO | httprunner.response:validate:209 - assert body.args.sum_v equal 3(str) ==> pass
2020-06-17 15:42:56.329 | INFO | httprunner.response:validate:209 - assert body.args.foo2 equal bar21(str) ==> pass
2020-06-17 15:42:56.330 | INFO | httprunner.runner:__run_step:290 - run step end: get with params <<<<<<
<Omit>
2020-06-17 15:42:57.019 | INFO | httprunner.runner:test_start:444 - generate testcase log: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/logs/cc404c49-000f-485c-b4c1-ac3367a053fe.run.log
.
======================================================================= 1 passed in 2.13s =======================================================================
每个用例都会生成一个执行log,位置:<ProjectRootDir>/logs/TestCaseID.run.log.
测试用例ID和Request ID
为排除故障,每个测试用例将生成唯一ID(uuid4),每个request header将自动添加一个带有测试用例ID的HRUN-Request-ID
HRUN-Request-ID = "HRUN-<TestCase ID>-<timestamp_six_digits>"
timestamp_six_digits = str(int(time.time() * 1000))[-6:])
换句话说,同一个测试用例中的所有request有同样的HRUN-Request-ID
前缀,并且每个请求有唯一的HRUN-Request-ID
后缀。
Client & Server IP:PORT
有时,记录远程服务器的IP和端口对于解决问题特别有用,尤其是当有多台服务器并且我们要检查哪一台返回错误时。
从3.0.13开始,HttpRunner将在调试阶段记录客户端和服务端IP:PORT
测试用例摘要:
"address": {
"client_ip": "10.90.205.63",
"client_port": 62802,
"server_ip": "34.233.204.163",
"server_port": 443
},
v2.x兼容性的参数
除了pytest的所有参数外,hrun还具有其他几个参数以保持与HttpRunner v2.x的兼容性。
--failfast:无效,此参数将自动删除 --report-file:指定html报告文件路径,此参数将替换为--html --self-contained-html并使用pytest-html插件生成html报告 --save-tests:如果设置,HttpRunner v3.x将创建一个pytest conftest.py文件,该文件包含会话固定装置以聚合每个测试用例的摘要并将其转储到summary.json