👏 🤷🏾 🤚🏼 GitLab Shell Runner。使用Docker Compose竞争性地启动测试服务 👨‍👧‍👧 👃🏼 💾

本文将同时引起测试人员和开发人员的兴趣，但更多地针对那些在基础设施资源不足和/或缺乏容器编排平台的情况下配置GitLab CI / CD进行集成测试的自动化人员。我将告诉您如何使用docker compose在单个GitLab Shell运行程序上配置测试环境的部署，以便在部署多个环境时，启动的服务不会相互干扰。

目录内容

背景知识

在我的实践中，通常会“修复”项目上的集成测试。通常第一个也是最重要的问题是CI管道，其中对已开发服务的集成测试是在开发 /阶段环境中进行的。这引起了很多问题：
- 由于集成测试过程中特定服务的缺陷，测试电路可能会因数据损坏而损坏。在某些情况下，发送带有损坏的JSON格式的请求时，服务会挂起，这使支架完全无法使用。
- 随着测试数据的增长减慢测试循环。我认为描述清理/回滚数据库的示例没有任何意义。在我的实践中，我还没有看到该过程顺利进行的项目。
- 在测试常规系统设置时，存在破坏测试电路性能的风险。例如，用户/组/密码/应用程序策略。
- 来自自动测试的测试数据阻止了手动测试人员的生存。
有人会说好的自动测试应该自己清除数据。我反对：
- 动态支架非常方便使用。
- 并非所有对象都可以通过API从系统中删除。例如，未实现删除对象的调用，因为它与业务逻辑相矛盾。
- 通过API创建对象时，会创建大量的元数据，这很难删除。
- 如果测试是相互依存的，那么在测试完成之后的数据清理过程将变得令人头疼。
- 对API的其他调用（在我看来，这是没有道理的）。
- 主要论点是：何时开始直接从数据库中清除测试数据。变成了真正的PK / FK马戏团！从开发人员那里可以听到：“我仅添加/删除/重命名了铭牌，为什么100500集成测试失败？”
我认为，最佳的解决方案是动态环境。
许多人使用docker-compose运行测试环境，但在CI / CD中进行集成测试时很少使用docker-compose。在这里，我不考虑kubernetes，集群和其他容器编排平台。并非每个公司都有它们。如果docker-compose.yml是通用的，那就太好了。
即使我们拥有自己的QA运行程序，我们如何确保通过docker-compose启动的服务不会互相干扰？
如何收集经过测试的服务的日志？
如何清洁转轮？

我有自己的项目GitLab运行程序，在为TestRail开发Java客户端时遇到了这些问题。确切地说，在运行集成测试时。在下文中，我们将以该项目中的示例解决这些问题。

到内容

Gitlab的壳赛跑者

对于奔跑者，我建议使用具有4个vCPU，4 GB RAM，50 GB HDD的Linux虚拟机。
在Internet上，有很多有关配置gitlab-runner的信息，因此简要介绍一下：

我们去SSH上的机器
如果您的RAM少于8 GB，那么我建议进行10 GB的交换，以免OOM杀手come来，并且由于缺少RAM而不会杀死我们。同时启动五个以上任务时，可能会发生这种情况。任务将较慢，但很稳定。
OOM杀手example
如果您看到bash: line 82: 26474 Killed在任务日志中被bash: line 82: 26474 Killed ，则只需运行sudo dmesg | grep 26474 sudo dmesg | grep 26474
```
 [26474] 1002 26474 1061935 123806 339 0 0 java Out of memory: Kill process 26474 (java) score 127 or sacrifice child Killed process 26474 (java) total-vm:4247740kB, anon-rss:495224kB, file-rss:0kB, shmem-rss:0kB 
```
如果图片看起来像这样，则添加交换或删除RAM。

安装gitlab-runner ， docker ， docker- compose ，make。

将gitlab-runner用户添加到gitlab-runner组

 sudo groupadd docker sudo usermod -aG docker gitlab-runner

注册 gitlab-runner。
打开以编辑/etc/gitlab-runner/config.toml并添加
```
 concurrent=20 [[runners]] request_concurrency = 10 
```
这将使您可以在一个运行程序上运行并行任务。在这里阅读更多。
如果您的计算机功能更强大，例如8个vCPU，16 GB RAM，那么这些数字至少可以增加2倍。但是，这完全取决于要在此跑步者上确切投放的数量和数量。

够了

到内容

准备docker-compose.yml

主要任务是docker-compose.yml，它将在本地和CI管道中使用。

变量COMPOSE_PROJECT_NAME将用于启动环境的多个实例（请参见makefile ）。

我的docker-compose.yml的示例

 version: "3" #    web (php)  fmt , #      . #   ,   /var/www/testrail volumes: static-content: services: db: image: mysql:5.7.22 environment: MYSQL_HOST: db MYSQL_DATABASE: mydb MYSQL_ROOT_PASSWORD: 1234 SKIP_GRANT_TABLES: 1 SKIP_NETWORKING: 1 SERVICE_TAGS: dev SERVICE_NAME: mysql migration: image: registry.gitlab.com/touchbit/image/testrail/migration:latest links: - db depends_on: - db fpm: image: registry.gitlab.com/touchbit/image/testrail/fpm:latest container_name: "testrail-fpm-${CI_JOB_ID:-local}" volumes: - static-content:/var/www/testrail links: - db web: image: registry.gitlab.com/touchbit/image/testrail/web:latest #   TR_HTTP_PORT  TR_HTTPS_PORTS  , #     80  443  . ports: - ${TR_HTTP_PORT:-80}:80 - ${TR_HTTPS_PORT:-443}:443 volumes: - static-content:/var/www/testrail links: - db - fpm

到内容

Makefile准备

我使用Makefile，因为它对于环境的本地管理和CI都很方便。

内嵌更多评论

 #           `.indirect`, #     `docker-compose.yml` #  bash   pipefail # pipefail -   ,      SHELL=/bin/bash -o pipefail #   CI_JOB_ID   ifeq ($(CI_JOB_ID),) #   local CI_JOB_ID := local endif #    export COMPOSE_PROJECT_NAME = $(CI_JOB_ID)-testrail #    , , volumes docker-down: docker-compose -f .indirect/docker-compose.yml down #   docker-down () docker-up: docker-down #     docker-registry docker-compose -f .indirect/docker-compose.yml pull #   # force-recreate -    # renew-anon-volumes -   volumes   docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d #  ,   ,           docker ps #    docker-logs: mkdir -p ./logs docker logs $${COMPOSE_PROJECT_NAME}_web_1 >& logs/testrail-web.log || true docker logs $${COMPOSE_PROJECT_NAME}_fpm_1 >& logs/testrail-fpm.log || true docker logs $${COMPOSE_PROJECT_NAME}_migration_1 >& logs/testrail-migration.log || true docker logs $${COMPOSE_PROJECT_NAME}_db_1 >& logs/testrail-mysql.log || true #   docker-clean: @echo   testrail- docker kill $$(docker ps --filter=name=testrail -q) || true @echo    docker rm -f $$(docker ps -a -f --filter=name=testrail status=exited -q) || true @echo  dangling  docker rmi -f $$(docker images -f "dangling=true" -q) || true @echo  testrail  docker rmi -f $$(docker images --filter=reference='registry.gitlab.com/touchbit/image/testrail/*' -q) || true @echo    volume docker volume rm -f $$(docker volume ls -q) || true @echo   testrail  docker network rm $(docker network ls --filter=name=testrail -q) || true docker ps

检查本地启动

 $ make docker-up docker-compose -f .indirect/docker-compose.yml pull Pulling db ... done Pulling migration ... done Pulling fpm ... done Pulling web ... done docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d Creating network "local-testrail_default" with the default driver Recreating local-testrail_db_1 ... done Recreating local-testrail_migration_1 ... done Recreating local-testrail_fpm_1 ... done Recreating local-testrail_web_1 ... done docker ps CONTAINER ID NAMES 3b8f9d4af29c local-testrail_web_1 5622c7d742d5 local-testrail_fpm_1 b580e3392038 local-testrail_migration_1 e467630bd3a5 local-testrail_db_1

检查CI启动

 $ export CI_JOB_ID=123456789 $ make docker-up docker-compose -f .indirect/docker-compose.yml pull Pulling db ... done Pulling migration ... done Pulling fpm ... done Pulling web ... done docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d Creating network "123456789-testrail_default" with the default driver Creating volume "123456789-testrail_static-content" with default driver Creating 123456789-testrail_db_1 ... done Creating 123456789-testrail_fpm_1 ... done Creating 123456789-testrail_migration_1 ... done Creating 123456789-testrail_web_1 ... done docker ps CONTAINER ID NAMES ccf1ad33d0e8 123456789-testrail_web_1 bc079964f681 123456789-testrail_fpm_1 10dc9d4d8f2a 123456789-testrail_migration_1 fe98d43c380e 123456789-testrail_db_1

检查日志收集

 $ make docker-logs mkdir -p ./logs docker logs ${COMPOSE_PROJECT_NAME}_web_1 >& logs/testrail-web.log || true docker logs ${COMPOSE_PROJECT_NAME}_fpm_1 >& logs/testrail-fpm.log || true docker logs ${COMPOSE_PROJECT_NAME}_migration_1 >& logs/testrail-migration.log || true docker logs ${COMPOSE_PROJECT_NAME}_db_1 >& logs/testrail-mysql.log || true

到内容

准备.gitlab-ci.yml

运行集成测试

 Integration: stage: test tags: - my-shell-runner before_script: #   registry - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} #   TR_HTTP_PORT  TR_HTTPS_PORT - export TR_HTTP_PORT=$(shuf -i10000-60000 -n1) - export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1) script: #    - make docker-up #    jar (  ) - java -jar itest.jar --http-port ${TR_HTTP_PORT} --https-port ${TR_HTTPS_PORT} #    - docker run --network=testrail-network-${CI_JOB_ID:-local} --rm itest after_script: #   - make docker-logs #   - make docker-down artifacts: #   when: always paths: - logs expire_in: 30 days

由于在工件中启动了这样的任务，logs目录将包含服务和测试的日志。如果发生错误，这非常方便。对我来说，每个并行测试都会写入自己的日志，但是我将分别讨论。

到内容

转轮清洗

该任务将仅按计划运行。

 stages: - clean - build - test Clean runner: stage: clean only: - schedules tags: - my-shell-runner script: - make docker-clean

接下来，转到我们的GitLab项目-> CI / CD->时间表->新时间表，然后添加新时间表

到内容

结果

在GitLab CI中运行4个任务

在带有集成测试的最后一个任务的日志中，我们看到了来自不同任务的容器

 CONTAINER ID NAMES c6b76f9135ed 204645172-testrail-web_1 01d303262d8e 204645172-testrail-fpm_1 2cdab1edbf6a 204645172-testrail-migration_1 826aaf7c0a29 204645172-testrail-mysql_1 6dbb3fae0322 204645084-testrail-web_1 3540f8d448ce 204645084-testrail-fpm_1 70fea72aa10d 204645084-testrail-mysql_1 d8aa24b2892d 204644881-testrail-web_1 6d4ccd910fad 204644881-testrail-fpm_1 685d8023a3ec 204644881-testrail-mysql_1 1cdfc692003a 204644793-testrail-web_1 6f26dfb2683e 204644793-testrail-fpm_1 029e16b26201 204644793-testrail-mysql_1 c10443222ac6 204567103-testrail-web_1 04339229397e 204567103-testrail-fpm_1 6ae0accab28d 204567103-testrail-mysql_1 b66b60d79e43 204553690-testrail-web_1 033b1f46afa9 204553690-testrail-fpm_1 a8879c5ef941 204553690-testrail-mysql_1 069954ba6010 204553539-testrail-web_1 ed6b17d911a5 204553539-testrail-fpm_1 1a1eed057ea0 204553539-testrail-mysql_1

更详细的日志

 $ docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded $ export TR_HTTP_PORT=$(shuf -i10000-60000 -n1) $ export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1) $ mkdir ${CI_JOB_ID} $ cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml $ make docker-up docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill docker network rm testrail-network-${CI_JOB_ID:-local} || true Error: No such network: testrail-network-204645172 docker network create testrail-network-${CI_JOB_ID:-local} 0a59552b4464b8ab484de6ae5054f3d5752902910bacb0a7b5eca698766d0331 docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull Pulling web ... done Pulling fpm ... done Pulling migration ... done Pulling db ... done docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d Creating volume "204645172-testrail_static-content" with default driver Creating 204645172-testrail-mysql_1 ... Creating 204645172-testrail-mysql_1 ... done Creating 204645172-testrail-migration_1 ... done Creating 204645172-testrail-fpm_1 ... done Creating 204645172-testrail-web_1 ... done docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c6b76f9135ed registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 13 seconds ago Up 1 second 0.0.0.0:51148->80/tcp, 0.0.0.0:25426->443/tcp 204645172-testrail-web_1 01d303262d8e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 16 seconds ago Up 13 seconds 9000/tcp 204645172-testrail-fpm_1 2cdab1edbf6a registry.gitlab.com/touchbit/image/testrail/migration:latest "docker-entrypoint.s…" 16 seconds ago Up 13 seconds 3306/tcp, 33060/tcp 204645172-testrail-migration_1 826aaf7c0a29 mysql:5.7.22 "docker-entrypoint.s…" 18 seconds ago Up 16 seconds 3306/tcp 204645172-testrail-mysql_1 6dbb3fae0322 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 36 seconds ago Up 22 seconds 0.0.0.0:44202->80/tcp, 0.0.0.0:20151->443/tcp 204645084-testrail-web_1 3540f8d448ce registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 38 seconds ago Up 35 seconds 9000/tcp 204645084-testrail-fpm_1 70fea72aa10d mysql:5.7.22 "docker-entrypoint.s…" 40 seconds ago Up 37 seconds 3306/tcp 204645084-testrail-mysql_1 d8aa24b2892d registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up 53 seconds 0.0.0.0:31103->80/tcp, 0.0.0.0:43872->443/tcp 204644881-testrail-web_1 6d4ccd910fad registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp 204644881-testrail-fpm_1 685d8023a3ec mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp 204644881-testrail-mysql_1 1cdfc692003a registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up About a minute 0.0.0.0:44752->80/tcp, 0.0.0.0:23540->443/tcp 204644793-testrail-web_1 6f26dfb2683e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp 204644793-testrail-fpm_1 029e16b26201 mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp 204644793-testrail-mysql_1 c10443222ac6 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:57123->80/tcp, 0.0.0.0:31657->443/tcp 204567103-testrail-web_1 04339229397e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp 204567103-testrail-fpm_1 6ae0accab28d mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp 204567103-testrail-mysql_1 b66b60d79e43 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:56321->80/tcp, 0.0.0.0:58749->443/tcp 204553690-testrail-web_1 033b1f46afa9 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp 204553690-testrail-fpm_1 a8879c5ef941 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp 204553690-testrail-mysql_1 069954ba6010 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:32869->80/tcp, 0.0.0.0:16066->443/tcp 204553539-testrail-web_1 ed6b17d911a5 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp 204553539-testrail-fpm_1 1a1eed057ea0 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp 204553539-testrail-mysql_1

所有任务成功完成

任务工件包含服务和测试日志