跳转至

运行测试用例

测试用例准备好后,运行hrun执行测试用例。 注意:hrunhttprunner run的缩写,他们功能一样。

hrun = httprunner run

运行用例的各种方式

(1)运行指定路径的用例

hrun path/to/testcase1

(2)指定多个测试用例路径执行

hrun path/to/testcase1 path/to/testcase2

(3)运行测试项目中所有测试用例 指定测试项目的绝对路径

hrun path/to/testcase_folder/

运行YAML/JSON格式的测试用例

如果测试用例格式是YAML或JSON,hrun先将YAML/JSON转为pytest(python)格式,然后再运行pytest。 相当于:

hrun = make + pytest

对于大多数用例,生成的pytest文件和YAML/JSON保存在相同目录下。生成的pytest用例名增加_test后缀,且.yml/yaml/.json替换为.py

/path/to/example.yml => /path/to/example_test.py

然而,如果测试用例目录名或用例文件名包含. - 空格这些字符,这些字符将被_替换,以避免python类中用例引用时的语法错误。 而且,以数字开头的目录和文件名,将添加T前缀,因为python模块和类名不能以数字开头。

path 1/a.b-2/3.yml => path_1/a_b_2/T3_test.py

运行pytest用例

当想运行pytest格式用例 或 YAML/JSON转换后的pytest用例,hrunpytest都可以使用。 需要记住的是,hrun只包装了pytest,所以pytest的功能都可以在hrun中使用。

$ hrun -h
usage: hrun [options] [file_or_dir] [file_or_dir] [...]

positional arguments:
  file_or_dir

general:
  -k EXPRESSION         仅运行与给定字符串表达式匹配的测试。表达式是python可识别的表达式,其中所有名称均为与测试文件名称及其父类进行子字符串匹配。
                        例如:`-k 'test_method or test_other'` 匹配所有测试方法名和类名包含'test_method' 或 'test_other'的用例。
                             `-k 'not test_method'` 匹配名称中不包含'test_method'的类。
                             `-k 'not test_method and not test_other'` 消除匹配项。  
                        另外,还运行,在其他的关键字匹配中,类和方法名成包含的,以及直接为其分配名称的函数。
                        匹配不区分大小写。

  -m MARKEXPR           只运行匹配给定表达式匹配的用例。 例如: -m 'mark1 and not mark2'.
  --markers             显示标记(内置标记,插件标记和只运行一次标记)
  -x, --exitfirst       遇到error或fail test立即退出.
  --maxfail=num         在第一个失败用例或错误后退出。
  --strict-markers, --strict
                        未在配置文件中"标记"部分注册的标记会引发错误。
  -c file              从'file'加载配置,而不是加载隐藏的配置文件。
  --continue-on-collection-errors
                        即使发生错误,也强制继续执行测试。
  --rootdir=ROOTDIR     定义测试项目的root目录. 可以是相对路径:'root_dir', './root_dir', 'root_dir/another_dir/'; 绝对路径:
                        '/home/user/root_dir'; 包含变量的路径: '$HOME/root_dir'.
  --fixtures, --funcargs
                        显示可用的fixtures, 按fixture名称排序 (以`_`开头的fixture,只有添加`-v`时才显示)
  --fixtures-per-test   每次测试都显示fixtures
  --import-mode={prepend,append}
                        当import测试模块时, prepend/append到系统环境变量中,默认前置到系统环境变量中。
  --pdb                 当出现error或强制断开时,启动交互式python调试器.
  --pdbcls=modulename:classname
                        针对错误启动自定义交互式python调试器. 例如: --pdbcls=IPython.terminal.debugger:TerminalPdb
  --trace               每次运行测试时立即跳出
  --capture=method      每次测试都捕获方法: fd|sys|no|tee-sys的其中之一。
  -s                    --capture=no的快捷模式.
  --runxfail            报告xfail标记的测试结果,就像没标记过一样。
  --lf, --last-failed   仅重新运行最后一次运行失败的测试(如果没有失败的则全部重新运行)
  --ff, --failed-first  运行所有测试,但先运行最后的失败用例。这可能会导致setup/teardown的重复运行。
  --nf, --new-first     首先运行新的测试用例,然后其余测试用例按照最后一次修改时间顺序执行。
  --cache-show=[CACHESHOW]
                        显示缓存内容, 不执行搜集或测试. 可选参数: glob (default: '*').
  --cache-clear         测试运行前清空所有缓存.
  --lfnf={all,none}, --last-failed-no-failures={all,none}
                        运行没有失败过的用例.
  --sw, --stepwise      测试失败时退出,且下次运行从最后失败的用例开始运行
  --stepwise-skip       忽略第一个失败的测试,下一个失败的用例出现时停止
  --allure-severities=SEVERITIES_SET
                        以逗号分割的优先级名称列表. 只运行指定优先级的用例. 可选择的优先级为: blocker, critical, normal, minor, trivial.
  --allure-epics=EPICS_SET
                        以逗号分割的epic列表. 运行具有至少一个指定标签的测试。
  --allure-features=FEATURES_SET
                        以逗号分割的feature列表. 运行具有至少一个指定标签的测试。
  --allure-stories=STORIES_SET
                        以逗号分割的story列表. 运行具有至少一个指定story的测试.
  --allure-link-pattern=LINK_TYPE:LINK_PATTERN
                        执行命令时指定短链接, 比如Bug地址. 格式为python str.format()格式,例如:`pytest test.py ----allure-link-pattern=issue:http//www.mytesttracker.com/issue/{}`.

reporting:
  --durations=N         显示最慢的N个测试用例和setup (N=0 for all).
  -v, --verbose         增加报告信息.
  -q, --quiet           减少报告信息.
  --verbosity=VERBOSE   设置详细程度,默认是0。
  -r chars              显示字符指定的测试摘要信息: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed
                        (a)ll except passed (p/P), or (A)ll. 默认启用(w)arnings (see --disable-warnings), 'N' 用于重置列表
                        (默认: 'fE').
  --disable-warnings, --disable-pytest-warnings
                        禁用警告摘要
  -l, --showlocals      在回溯中显示局部变量(默认禁止).
  --tb=style            追溯打印模式 (auto/long/short/line/native/no).
  --show-capture={no,stdout,stderr,log,all}
                        控制在失败的测试中如何显示捕获的stdout/stderr/log. 默认'all'.
  --full-trace          不删减任何信息(默认是删减的).
  --color=color         彩色终端输出 (yes/no/auto).
  --pastebin=mode       发送失败|all信息到bpaste.net服务
  --junit-xml=path      在指定路径生成junit-xml报告
  --junit-prefix=str    在junit-xml输出中添加str到类名前面
  --result-log=path     可读结果日志的已弃用路径
  --html=path           指定路径生成html报告
  --self-contained-html
                        创建一个包含所有必需的样式,脚本和图像的自包含html文件 - 这意味着该报告可能不会呈现或具备CSP限制的功能 (见: https://developer.mozilla.org/docs/Web/Security/CSP)
  --css=path            将给定的CSS文件内容附加到报表样式文件中。

collection:
  --collect-only, --co  仅收集测试,不执行测试。
  --pyargs              尝试将所有参数解释为python软件包。
  --ignore=path         收集期间忽略指定路径(允许多个路径).
  --ignore-glob=path    收集期间忽略路径 (允许多个路径).
  --deselect=nodeid_prefix
                        在收集期间取消选择项(通过节点id前缀)(允许多个)
  --confcutdir=dir      仅加载指定目录(conftest.py的相对路径)
  --noconftest          不加载任何conftest.py files.
  --keep-duplicates     保持重复测试
  --collect-in-virtualenv
                        不要忽略本地virtualenv目录中的测试
  --doctest-modules     在所有.py模块中运行doctests
  --doctest-report={none,cdiff,ndiff,udiff,only_first_failure}
                        为doctest失败时的差异选择另一种输出格式
  --doctest-glob=pat    doctests文件匹配模式,默认值:test * .txt
  --doctest-ignore-import-errors
                        忽略 doctest ImportErrors
  --doctest-continue-on-failure
                        对于给定的doctest,在第一次失败后继续运行

测试会话调试和配置:
  --basetemp=dir        此测试运行的基本临时目录。(警告:如果存在此目录,则将其删除)
  -V, --version         显示pytest版本和有关插件的信息。
  -h, --help            显示帮助消息和配置信息
  -p name               尽早加载给定的插件模块名称或入口点(允许多个). 为避免加载插件, 使用`no:` prefix, 例如. `no:doctest`.
  --trace-config        跟踪conftest.py文件的注意事项。
  --debug               将内部跟踪调试信息存储在“ pytestdebug.log”中。
  -o OVERRIDE_INI, --override-ini=OVERRIDE_INI
                        以“option = value”样式覆盖ini选项,例如 `-o xfail_strict = True -o cache_dir = cache`
  --assert=MODE         控制断言调试工具。 'plain'不执行任何断言调试。 “ rewrite”(默认)在导入时重写测试模块中的assert语句,以提供assert表达式信息。
  --setup-only          仅安装fixtures, 不执行测试.
  --setup-show          执行测试时显示fixtures的setup.
  --setup-plan          不执行,只显示将要执行的fixture和测试用例

pytest-warnings:
  -W PYTHONWARNINGS, --pythonwarnings=PYTHONWARNINGS
                        set which warnings to report, see -W option of python itself.

logging:
  --no-print-logs       disable printing caught logs on failed tests.
  --log-level=LEVEL     level of messages to catch/display. Not set by default, so it depends on the root/parent log handler's effective level, where it is
                        "WARNING" by default.
  --log-format=LOG_FORMAT
                        log format as used by the logging module.
  --log-date-format=LOG_DATE_FORMAT
                        log date format as used by the logging module.
  --log-cli-level=LOG_CLI_LEVEL
                        cli logging level.
  --log-cli-format=LOG_CLI_FORMAT
                        log format as used by the logging module.
  --log-cli-date-format=LOG_CLI_DATE_FORMAT
                        log date format as used by the logging module.
  --log-file=LOG_FILE   path to a file when logging will be written to.
  --log-file-level=LOG_FILE_LEVEL
                        log file logging level.
  --log-file-format=LOG_FILE_FORMAT
                        log format as used by the logging module.
  --log-file-date-format=LOG_FILE_DATE_FORMAT
                        log date format as used by the logging module.
  --log-auto-indent=LOG_AUTO_INDENT
                        Auto-indent multiline messages passed to the logging module. Accepts true|on, false|off or an integer.

reporting:
  --alluredir=DIR       Generate Allure report in the specified directory (may not exist)
  --clean-alluredir     Clean alluredir folder if it exists
  --allure-no-capture   Do not attach pytest captured logging/stdout/stderr to report

custom options:
  --metadata=key value  additional metadata.
  --metadata-from-json=METADATA_FROM_JSON
                        additional metadata from a json string.

[pytest] ini-options in the first pytest.ini|tox.ini|setup.cfg file found:

  markers (linelist):   markers for test functions
  empty_parameter_set_mark (string):
                        default marker for empty parametersets
  norecursedirs (args): directory patterns to avoid for recursion
  testpaths (args):     directories to search for tests when no files or directories are given in the command line.
  usefixtures (args):   list of default fixtures to be used with this project
  python_files (args):  glob-style file patterns for Python test module discovery
  python_classes (args):
                        prefixes or glob names for Python test class discovery
  python_functions (args):
                        prefixes or glob names for Python test function and method discovery
  disable_test_id_escaping_and_forfeit_all_rights_to_community_support (bool):
                        disable string escape non-ascii characters, might cause unwanted side effects(use at your own risk)
  console_output_style (string):
                        console output: "classic", or with additional progress information ("progress" (percentage) | "count").
  xfail_strict (bool):  default for the strict parameter of xfail markers when not given explicitly (default: False)
  enable_assertion_pass_hook (bool):
                        Enables the pytest_assertion_pass hook.Make sure to delete any previously generated pyc cache files.
  junit_suite_name (string):
                        Test suite name for JUnit report
  junit_logging (string):
                        Write captured log messages to JUnit report: one of no|log|system-out|system-err|out-err|all
  junit_log_passing_tests (bool):
                        Capture log information for passing tests to JUnit report:
  junit_duration_report (string):
                        Duration time to report: one of total|call
  junit_family (string):
                        Emit XML for schema: one of legacy|xunit1|xunit2
  doctest_optionflags (args):
                        option flags for doctests
  doctest_encoding (string):
                        encoding used for doctest files
  cache_dir (string):   cache directory path.
  filterwarnings (linelist):
                        Each line specifies a pattern for warnings.filterwarnings. Processed after -W/--pythonwarnings.
  log_print (bool):     default value for --no-print-logs
  log_level (string):   default value for --log-level
  log_format (string):  default value for --log-format
  log_date_format (string):
                        default value for --log-date-format
  log_cli (bool):       enable log display during test run (also known as "live logging").
  log_cli_level (string):
                        default value for --log-cli-level
  log_cli_format (string):
                        default value for --log-cli-format
  log_cli_date_format (string):
                        default value for --log-cli-date-format
  log_file (string):    default value for --log-file
  log_file_level (string):
                        default value for --log-file-level
  log_file_format (string):
                        default value for --log-file-format
  log_file_date_format (string):
                        default value for --log-file-date-format
  log_auto_indent (string):
                        default value for --log-auto-indent
  faulthandler_timeout (string):
                        Dump the traceback of all threads if a test takes more than TIMEOUT seconds to finish. Not available on Windows.
  addopts (args):       extra command line options
  minversion (string):  minimally required pytest version
  render_collapsed (bool):
                        Open the report with all rows collapsed. Useful for very large reports

environment variables:
  PYTEST_ADDOPTS           extra command line options
  PYTEST_PLUGINS           comma-separated plugins to load during startup
  PYTEST_DISABLE_PLUGIN_AUTOLOAD set to disable plugin auto-loading
  PYTEST_DEBUG             set to enable debug tracing of pytest's internals


to see available markers type: pytest --markers
to see available fixtures type: pytest --fixtures
(shown according to specified file_or_dir or current dir if not specified; fixtures with leading '_' are only shown with the '-v' option

执行日志

$ hrun examples/postman_echo/request_methods/request_with_functions.yml 
2020-06-17 15:39:41.041 | INFO     | httprunner.make:make_testcase:317 - start to make testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:39:41.042 | INFO     | httprunner.make:make_testcase:390 - generated testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
2020-06-17 15:39:41.042 | INFO     | httprunner.make:format_pytest_with_black:154 - format pytest cases with black ...
reformatted /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
2020-06-17 15:39:41.315 | INFO     | httprunner.cli:main_run:56 - start to run tests with pytest. HttpRunner version: 3.0.13
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner
plugins: metadata-1.9.0, allure-pytest-2.8.16, html-2.1.1
collected 1 item                                                                                                                                                

examples/postman_echo/request_methods/request_with_functions_test.py .                                                                                    [100%]

======================================================================= 1 passed in 2.98s =======================================================================

如果你想看到request和response、提取和断言的详细信息,可以增加-s--capture=no的缩写)

$ hrun -s examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:42:54.369 | INFO     | httprunner.make:make_testcase:317 - start to make testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions.yml
2020-06-17 15:42:54.369 | INFO     | httprunner.make:make_testcase:390 - generated testcase: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
2020-06-17 15:42:54.370 | INFO     | httprunner.make:format_pytest_with_black:154 - format pytest cases with black ...
reformatted /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/request_methods/request_with_functions_test.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
2020-06-17 15:42:54.699 | INFO     | httprunner.cli:main_run:56 - start to run tests with pytest. HttpRunner version: 3.0.13
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner
plugins: metadata-1.9.0, allure-pytest-2.8.16, html-2.1.1
collected 1 item                                                                                                                                                

examples/postman_echo/request_methods/request_with_functions_test.py 2020-06-17 15:42:55.017 | INFO     | httprunner.runner:test_start:435 - Start to run testcase: request methods testcase with functions, TestCase ID: cc404c49-000f-485c-b4c1-ac3367a053fe
2020-06-17 15:42:55.018 | INFO     | httprunner.runner:__run_step:278 - run step begin: get with params >>>>>>
2020-06-17 15:42:56.326 | DEBUG    | httprunner.client:log_print:40 - 
================== request details ==================
method   : GET
url      : https://postman-echo.com/get?foo1=bar11&foo2=bar21&sum_v=3
headers  : {
    "User-Agent": "HttpRunner/3.0.13",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "HRUN-Request-ID": "HRUN-cc404c49-000f-485c-b4c1-ac3367a053fe-775018",
    "Content-Length": "2",
    "Content-Type": "application/json"
}
cookies  : {}
body     : {}

2020-06-17 15:42:56.327 | DEBUG    | httprunner.client:log_print:40 - 
================== response details ==================
status_code : 200
headers  : {
    "Date": "Wed, 17 Jun 2020 07:42:56 GMT",
    "Content-Type": "application/json; charset=utf-8",
    "Content-Length": "477",
    "Connection": "keep-alive",
    "ETag": "W/\"1dd-2JtBYPcnh8D6fqLz8KFn16Oq1R0\"",
    "Vary": "Accept-Encoding",
    "set-cookie": "sails.sid=s%3A6J_EtUk3nkL_C2xtx-NtAXrlA5wPxEgk.gIO2yBbtvGWIIgQ%2F2mZhMkU669G3F60cvLAPWbwyoGM; Path=/; HttpOnly"
}
cookies  : {
    "sails.sid": "s%3A6J_EtUk3nkL_C2xtx-NtAXrlA5wPxEgk.gIO2yBbtvGWIIgQ%2F2mZhMkU669G3F60cvLAPWbwyoGM"
}
encoding : utf-8
content_type : application/json; charset=utf-8
body     : {
    "args": {
        "foo1": "bar11",
        "foo2": "bar21",
        "sum_v": "3"
    },
    "headers": {
        "x-forwarded-proto": "https",
        "x-forwarded-port": "443",
        "host": "postman-echo.com",
        "x-amzn-trace-id": "Root=1-5ee9c980-d8e98cc72a26ef24f5819ce3",
        "content-length": "2",
        "user-agent": "HttpRunner/3.0.13",
        "accept-encoding": "gzip, deflate",
        "accept": "*/*",
        "hrun-request-id": "HRUN-cc404c49-000f-485c-b4c1-ac3367a053fe-775018",
        "content-type": "application/json"
    },
    "url": "https://postman-echo.com/get?foo1=bar11&foo2=bar21&sum_v=3"
}

2020-06-17 15:42:56.328 | INFO     | httprunner.client:request:203 - status_code: 200, response_time(ms): 1307.33 ms, response_length: 477 bytes
2020-06-17 15:42:56.328 | INFO     | httprunner.response:extract:152 - extract mapping: {'foo3': 'bar21'}
2020-06-17 15:42:56.328 | INFO     | httprunner.response:validate:209 - assert status_code equal 200(int)       ==> pass
2020-06-17 15:42:56.329 | INFO     | httprunner.response:validate:209 - assert body.args.foo1 equal bar11(str)  ==> pass
2020-06-17 15:42:56.329 | INFO     | httprunner.response:validate:209 - assert body.args.sum_v equal 3(str)     ==> pass
2020-06-17 15:42:56.329 | INFO     | httprunner.response:validate:209 - assert body.args.foo2 equal bar21(str)  ==> pass
2020-06-17 15:42:56.330 | INFO     | httprunner.runner:__run_step:290 - run step end: get with params <<<<<<

<Omit>

2020-06-17 15:42:57.019 | INFO     | httprunner.runner:test_start:444 - generate testcase log: /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/examples/postman_echo/logs/cc404c49-000f-485c-b4c1-ac3367a053fe.run.log
.

======================================================================= 1 passed in 2.13s =======================================================================

每个用例都会生成一个执行log,位置:<ProjectRootDir>/logs/TestCaseID.run.log.

测试用例ID和Request ID

为排除故障,每个测试用例将生成唯一ID(uuid4),每个request header将自动添加一个带有测试用例ID的HRUN-Request-ID

HRUN-Request-ID = "HRUN-<TestCase ID>-<timestamp_six_digits>"
timestamp_six_digits = str(int(time.time() * 1000))[-6:])

换句话说,同一个测试用例中的所有request有同样的HRUN-Request-ID前缀,并且每个请求有唯一的HRUN-Request-ID后缀。

Client & Server IP:PORT

有时,记录远程服务器的IP和端口对于解决问题特别有用,尤其是当有多台服务器并且我们要检查哪一台返回错误时。

从3.0.13开始,HttpRunner将在调试阶段记录客户端和服务端IP:PORT

测试用例摘要:

"address": {
    "client_ip": "10.90.205.63",
    "client_port": 62802,
    "server_ip": "34.233.204.163",
    "server_port": 443
},

v2.x兼容性的参数

除了pytest的所有参数外,hrun还具有其他几个参数以保持与HttpRunner v2.x的兼容性。

--failfast:无效,此参数将自动删除 --report-file:指定html报告文件路径,此参数将替换为--html --self-contained-html并使用pytest-html插件生成html报告 --save-tests:如果设置,HttpRunner v3.x将创建一个pytest conftest.py文件,该文件包含会话固定装置以聚合每个测试用例的摘要并将其转储到summary.json