diff options
author | 2020-11-16 21:01:42 +0100 | |
---|---|---|
committer | 2020-11-18 15:18:41 +0100 | |
commit | 939bacebe38cd02c47c0d87d513032320503acbf (patch) | |
tree | 442e436902ebe20526ebcdbcfd4377c380112003 | |
parent | Merge pull request #78 - Update to Python 3.9 (diff) |
Migrate build pipeline to Github Actions
I've migrated the build pipeline to GitHub Actions and changed the
container registry to GitHub Container Registry. In the process, I've
made some changes to our docker setup and caching:
- We are now using a single multi-stage Dockerfile
Instead of three separate dockerfiles, we are now using a
single multi-stage Dockerfile that can be used to build the three images
we want using build targets.
In part, this is because we're now using the docker buildx build action
currently recommended by docker. This new engine runs in a sandboxed
mode, meaning that while it can export built images to `docker` running
in the host, it cannot import local images from it to base builds on.
- Docker builds are now cached within GitHub Actions
The builds are now cached using the GitHub Actions cache of the build
cache directory. The cache keys try to match a cache generated by a
build that matches the current build as closely as possible. In case of
a cache miss, we fall back to caching from the latest image pushed to
the container repository.
- The `base` and `venv` images now have an inline cache manifest
In order to fall back intelligently to caching from the repository, the
final build and push action for the `base` and `venv` images includes an
"inline" cache manifest. This means that the build process can inspect,
without pulling, if it makes sense to pull layers to speed up the build.
The other options, pushing a cache manifest separately (not inline), is
currently not supported by GHCR.
The custom caching script has been removed.
- Linting errors are now added as GitHub Actions annotations
Just like for some of our other pipelines, linting now generates
annotations if linting errors are observed.
- Coverage is pushed to coveralls.io
A coverage summary is now pushed to coveralls.io. Each CI run will get a
unique job that's linked in the CI output. If the run is attached to a
PR, coveralls.io will automatically add a check link with the coverage
result to the PR as well.
- The README.md, Pipfile, docker-compose, and scripts have been updated
As we now need to pull from and link to the GHCR, I've updated the other
files to reflect these changes, including Pipfile run commands. I've
also changed the CI badge and added a coveralls.io badge.
-rw-r--r-- | .github/workflows/lint-test-build-push.yaml | 207 | ||||
-rw-r--r-- | Pipfile | 17 | ||||
-rw-r--r-- | README.md | 16 | ||||
-rw-r--r-- | azure-pipelines.yml | 20 | ||||
-rw-r--r-- | ci/build.yml | 51 | ||||
-rw-r--r-- | ci/lint-test.yml | 41 | ||||
-rw-r--r-- | ci/push.yml | 39 | ||||
-rw-r--r-- | ci/setup.yml | 23 | ||||
-rw-r--r-- | docker-compose.yml | 2 | ||||
-rw-r--r-- | docker/Dockerfile | 58 | ||||
-rw-r--r-- | docker/base.Dockerfile | 34 | ||||
-rw-r--r-- | docker/venv.Dockerfile | 20 | ||||
-rwxr-xr-x | scripts/check_dockerfiles.sh | 127 | ||||
-rwxr-xr-x | scripts/dev.sh | 9 |
14 files changed, 289 insertions, 375 deletions
diff --git a/.github/workflows/lint-test-build-push.yaml b/.github/workflows/lint-test-build-push.yaml new file mode 100644 index 0000000..641b4e6 --- /dev/null +++ b/.github/workflows/lint-test-build-push.yaml @@ -0,0 +1,207 @@ +name: Lint, Test, Build, Push + +on: + push: + branches: + - master + pull_request: + + +jobs: + lint-test: + runs-on: ubuntu-latest + env: + # Determine whether or not we should build the + # final production image and push it to GHCR. + production_build: ${{ github.event_name != 'pull_request' && + github.ref == 'refs/heads/sebastiaan/backend/cache-docker-images' }} + + steps: + # Create a short SHA-tag to tag built images + - name: Create SHA Container Tag + id: sha_tag + run: | + tag=$(cut -c 1-7 <<< $GITHUB_SHA) + echo "::set-output name=tag::$tag" + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 + + - name: Login to Github Container Registry + uses: docker/login-action@v1 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GHCR_TOKEN }} + + # Set up a caching directory for image layers. According to the docker + # documentation, it's recommended to use a SHA-based key to get the + # greatest change of finding the most relevant cached layer. We fall + # down to more generic containers by then matching by GitHub branch, + # to use cache generated earlier in the same branch, and finally to + # the latest cache in general. The `v0` is purely a cache version + # indicator that can be incremented manually if we want to invalidate + # old caches completely. + - name: Cache Image Layers + uses: actions/cache@v2 + with: + path: /tmp/.buildx-cache + key: ${{ runner.os }}-v0-buildx-${{ github.ref }}-${{ github.sha }} + restore-keys: | + ${{ runner.os }}-v0-buildx-${{ github.ref }}- + ${{ runner.os }}-v0-buildx- + + # Build the image we need for testing/linting the current codebase, + # without pushing the image to the GHCR. Instead, we load it into + # the runner's docker environment so we can run it later. The + # target of this build is the `venv` stage of the Dockerfile, as we + # don't want to include the final production entry point stage. + # + # This build caches to our GitHub Actions cache and uses that cache + # during the build process as well. If no GitHub Actions cache was + # available, it will use the latest intermediate images pushed to + # the GHCR as a cache source. + - name: Build image for linting and testing + uses: docker/build-push-action@v2 + with: + context: . + file: ./docker/Dockerfile + push: false + load: true + target: venv + cache-from: | + type=local,src=/tmp/.buildx-cache + ghcr.io/python-discord/snekbox-base:latest + ghcr.io/python-discord/snekbox-venv:latest + cache-to: type=local,dest=/tmp/.buildx-cache,mode=max + tags: ghcr.io/python-discord/snekbox-venv:${{ steps.sha_tag.outputs.tag }} + + - name: Start Container + run: "docker run \ + --tty \ + --detach \ + --name snekbox_test \ + --privileged \ + --hostname pdsnk-dev \ + -e PYTHONDONTWRITEBYTECODE=1 \ + -e PIPENV_PIPFILE='/snekbox/Pipfile\' \ + -e ENV=\"${PWD}/scripts/.profile\" \ + --volume \"${PWD}\":\"${PWD}\" \ + --workdir \"${PWD}\" \ + --entrypoint /bin/bash \ + ghcr.io/python-discord/snekbox-venv:${{ steps.sha_tag.outputs.tag }}" + + - name: Install dependencies + run: "docker exec snekbox_test /bin/bash -c \ + 'pipenv install --system --deploy --dev'" + + # This runs `flake8` in the container and asks `flake8` to output + # linting errors in the format of the command for registering workflow + # error messages/annotations. This means that Github Actions will pick + # up on this output to generate nice annotations to indicate what went + # wrong where. + - name: Run linter + run: "docker exec snekbox_test /bin/bash -c 'flake8 \ + --format \"::error file=%(path)s,line=%(row)d,col=%(col)d::\ + [flake8] %(code)s: %(text)s\"'" + + # Memory limit tests would fail if this isn't disabled. + - name: Disable swap memory + run: sudo swapoff -a + + # Run unittests and generate coverage report in the container + - name: Run unit tests and generate coverage report + id: run_tests + run: | + echo '::set-output name=started::true' + cmd='coverage run -m unittest; coverage report -m' + docker exec snekbox_test /bin/bash -c "${cmd}" + + # Set-up a Python version to process the coverage reports + # Note: This step runs even if the test step failed to make + # sure we process the coverage reports. + - name: Setup python + if: always() && steps.run_tests.outputs.started == 'true' + id: python + uses: actions/setup-python@v2 + with: + python-version: '3.9' + + # We'll only ever need a single dependency in this python + # environment and we'll only use it in the CI, so let's + # install it directly here and run it. + # + # This step will publish the coverage results to coveralls.io + # print a job link in the output. It will also register a + # step in the check suite visible in the PR with a link to + # the job. + - name: Publish coverage report to coveralls.io + if: always() && steps.run_tests.outputs.started == 'true' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + pip install coveralls~=2.1 + coveralls + + # Final build stage. This is run in the same job with conditions + # to prevent us from having to reload the caching directory. We + # already built a huge chunk of the image before this point in + # the run, so it does not make sense to drop down to a completely + # fresh build environment in a new worker/runner. + + # Build the final production image and push it to GHCR, tagging it + # both with the short commit SHA and 'latest'. This step should use + # the cache that was just generated when we built the test container. + - name: Build final image + if: env.production_build == 'true' + uses: docker/build-push-action@v2 + with: + context: . + file: ./docker/Dockerfile + push: true + cache-from: | + type=local,src=/tmp/.buildx-cache + ghcr.io/python-discord/snekbox-base:latest + ghcr.io/python-discord/snekbox-venv:latest + ghcr.io/python-discord/snekbox:latest + cache-to: type=local,dest=/tmp/.buildx-cache + tags: | + ghcr.io/python-discord/snekbox:latest + ghcr.io/python-discord/snekbox:${{ steps.sha_tag.outputs.tag }} + + # Push the base image to GHCR, *with* an inline cache manifest to + # ensure we can use this image as a cache source if our GitHub Actions + # "local" cache failed to be restored. GHCR does not support pushing a + # separate cache manifest, meaning we have to use an "inline" manifest. + - name: Push base image + if: env.production_build == 'true' + uses: docker/build-push-action@v2 + with: + context: . + file: ./docker/Dockerfile + target: base + push: true + cache-from: | + type=local,src=/tmp/.buildx-cache + ghcr.io/python-discord/snekbox-base:latest + cache-to: type=inline + tags: ghcr.io/python-discord/snekbox-base:latest + + # Push the venv image to GHCR *with* an inline cache manifest. See + # the comment attached to the previous step for more information. + - name: Push venv image + if: env.production_build == 'true' + uses: docker/build-push-action@v2 + with: + context: . + file: ./docker/Dockerfile + target: venv + push: true + cache-from: | + type=local,src=/tmp/.buildx-cache + ghcr.io/python-discord/snekbox-base:latest + ghcr.io/python-discord/snekbox-venv:latest + cache-to: type=inline + tags: ghcr.io/python-discord/snekbox-venv:latest @@ -61,26 +61,29 @@ snekbox = "gunicorn -c config/gunicorn.conf.py snekbox.api.app" devsh = "sh scripts/dev.sh" build = """ docker build \ - -t pythondiscord/snekbox:latest \ + -t ghcr.io/python-discord/snekbox:latest \ -f docker/Dockerfile \ . """ buildbase = """ docker build \ - -t pythondiscord/snekbox-base:latest \ - -f docker/base.Dockerfile \ + -t ghcr.io/python-discord/snekbox-base:latest \ + -f docker/Dockerfile \ + --target base \ . """ buildvenv = """ docker build \ - -t pythondiscord/snekbox-venv:latest \ - -f docker/venv.Dockerfile \ + -t ghcr.io/python-discord/snekbox-venv:latest \ + -f docker/Dockerfile \ + --target venv \ . """ builddev = """ docker build \ - -t pythondiscord/snekbox-venv:dev \ - -f docker/venv.Dockerfile \ + -t ghcr.io/python-discord/snekbox-venv:dev \ + -f docker/Dockerfile \ + --target venv \ --build-arg DEV=1 \ . """ @@ -1,4 +1,4 @@ -[![Build Status][1]][2] +[![Build Status][1]][2] [![Coverage Status][3]][4] # snekbox @@ -33,10 +33,10 @@ See [`snekapi.py`] and [`resources`] for API documentation. ## Running snekbox -A Docker image is available on [Docker Hub]. A container can be started with the following command, which will also pull the image if it doesn't currently exist locally: +A Docker image is available in the [GitHub Container Registry]. A container can be started with the following command, which will also pull the image if it doesn't currently exist locally: ``` -docker run --ipc=none --privileged -p 8060:8060 pythondiscord/snekbox +docker run --ipc=none --privileged -p 8060:8060 ghcr.io/python-discord/snekbox ``` To run it in the background, use the `-d` option. See the documentation on [`docker run`] for more information. @@ -47,7 +47,7 @@ The above command will make the API accessible on the host via `http://localhost ### Initial Setup -A Python 3.8 interpreter and the [pipenv] package are required. Once those requirements are satisfied, install the project's dependencies: +A Python 3.9 interpreter and the [pipenv] package are required. Once those requirements are satisfied, install the project's dependencies: ``` pipenv sync --dev @@ -142,8 +142,10 @@ nsjpy "print('hello world!')" The alias can be found in `./scripts/.profile`, which is automatically added when the shell is launched in the container. -[1]: https://dev.azure.com/python-discord/Python%20Discord/_apis/build/status/Snekbox?branchName=master -[2]: https://dev.azure.com/python-discord/Python%20Discord/_build/latest?definitionId=13&branchName=master +[1]: https://github.com/python-discord/snekbox/workflows/Lint,%20Test,%20Build,%20Push/badge.svg?branch=master +[2]: https://github.com/python-discord/snekbox/actions?query=workflow%3A%22Lint%2C+Test%2C+Build%2C+Push%22+branch%3Amaster +[3]: https://coveralls.io/repos/github/python-discord/snekbox/badge.svg?branch=master +[4]: https://coveralls.io/github/python-discord/snekbox?branch=master [`snekbox.cfg`]: config/snekbox.cfg [`snekapi.py`]: snekbox/api/snekapi.py [`resources`]: snekbox/api/resources @@ -151,5 +153,5 @@ The alias can be found in `./scripts/.profile`, which is automatically added whe [nsjail]: https://github.com/google/nsjail [falcon]: https://falconframework.org/ [gunicorn]: https://gunicorn.org/ -[docker hub]: https://hub.docker.com/r/pythondiscord/snekbox +[GitHub Container Registry]: https://github.com/orgs/python-discord/packages/container/package/snekbox [pipenv]: https://docs.pipenv.org/en/latest/ diff --git a/azure-pipelines.yml b/azure-pipelines.yml deleted file mode 100644 index 573e3cc..0000000 --- a/azure-pipelines.yml +++ /dev/null @@ -1,20 +0,0 @@ -# https://aka.ms/yaml - -jobs: - - job: build_lint_test_push - displayName: 'Build, Lint, Test, & Push' - - pool: - vmImage: 'ubuntu-18.04' - - variables: - BASE_CHANGED: 'True' - VENV_CHANGED: 'True' - BASE_PULL: 'False' - VENV_PULL: 'False' - - steps: - - template: ci/build.yml - - template: ci/setup.yml - - template: ci/lint-test.yml - - template: ci/push.yml diff --git a/ci/build.yml b/ci/build.yml deleted file mode 100644 index 7d51709..0000000 --- a/ci/build.yml +++ /dev/null @@ -1,51 +0,0 @@ -steps: - - task: ShellScript@2 - displayName: 'Check If Images Need to Be Built' - inputs: - scriptPath: scripts/check_dockerfiles.sh - disableAutoCwd: true - - # Without a login, the following Docker build steps wouldn't add image tags. - - task: Docker@1 - displayName: 'Log into Docker Hub' - inputs: - command: login - containerregistrytype: 'Container Registry' - dockerRegistryEndpoint: 'DockerHub' - - # Building the venv depends on this base image. Build the base if it can't - # pulled from Docker Hub, which will be the case if the base Dockerfile has - # has had changes. - - script: | - docker build \ - -f docker/base.Dockerfile \ - -t pythondiscord/snekbox-base:latest \ - . - displayName: 'Build Base Image' - condition: > - and( - succeeded(), - or( - eq(variables.BASE_CHANGED, 'True'), - and( - eq(variables.VENV_CHANGED, 'True'), - eq(variables.BASE_PULL, 'False') - ) - ) - ) - - # Build the venv image if it's had changes or it can't be pulled. - - script: | - docker build \ - -f docker/venv.Dockerfile \ - -t pythondiscord/snekbox-venv:latest \ - . - displayName: 'Build Virtual Environment Image' - condition: > - and( - succeeded(), - or( - eq(variables.VENV_CHANGED, 'True'), - eq(variables.VENV_PULL, 'False') - ) - ) diff --git a/ci/lint-test.yml b/ci/lint-test.yml deleted file mode 100644 index 2d70f6e..0000000 --- a/ci/lint-test.yml +++ /dev/null @@ -1,41 +0,0 @@ -steps: - - script: | - docker exec snekbox_test /bin/bash -c \ - 'flake8; flake8 --format junit-xml --output-file test-lint.xml' - displayName: 'Run Linter' - - - task: PublishTestResults@2 - displayName: 'Publish Lint Results' - condition: succeededOrFailed() - inputs: - testResultsFiles: '**/test-lint.xml' - testRunTitle: 'Lint Results' - - # Memory limit tests would fail if this isn't disabled. - - script: sudo swapoff -a - displayName: 'Disable Swap Memory' - - - script: | - docker exec snekbox_test /bin/bash -c \ - 'coverage run -m xmlrunner' - displayName: 'Run Unit Tests' - - - task: PublishTestResults@2 - displayName: 'Publish Test Results' - condition: succeededOrFailed() - inputs: - testResultsFiles: '**/TEST-*.xml' - testRunTitle: 'Test Results' - - # Run report too because the XML report doesn't output to stdout. - - script: | - docker exec snekbox_test /bin/bash -c \ - 'coverage report && coverage xml' - displayName: 'Generate Coverage Report' - - - task: PublishCodeCoverageResults@1 - displayName: 'Publish Coverage Results' - condition: succeededOrFailed() - inputs: - codeCoverageTool: Cobertura - summaryFileLocation: '**/coverage.xml' diff --git a/ci/push.yml b/ci/push.yml deleted file mode 100644 index 9449df0..0000000 --- a/ci/push.yml +++ /dev/null @@ -1,39 +0,0 @@ -steps: - # Always build this image unless it's for a pull request. - - script: | - docker build \ - -f docker/Dockerfile \ - -t pythondiscord/snekbox:latest \ - . - displayName: 'Build Final Image' - condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest')) - - # Push images only after they've all successfully been built. - - script: docker push pythondiscord/snekbox-base:latest - displayName: 'Push Base Image' - condition: > - and( - succeeded(), - ne(variables['Build.Reason'], 'PullRequest'), - ne(variables.BASE_PULL, 'True'), - or( - eq(variables.BASE_CHANGED, 'True'), - eq(variables.VENV_CHANGED, 'True') - ) - ) - - - script: docker push pythondiscord/snekbox-venv:latest - displayName: 'Push Virtual Environment Image' - condition: > - and( - succeeded(), - ne(variables['Build.Reason'], 'PullRequest'), - or( - eq(variables.BASE_CHANGED, 'True'), - eq(variables.VENV_CHANGED, 'True') - ) - ) - - - script: docker push pythondiscord/snekbox:latest - displayName: 'Push Final Image' - condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest')) diff --git a/ci/setup.yml b/ci/setup.yml deleted file mode 100644 index 1c66aa7..0000000 --- a/ci/setup.yml +++ /dev/null @@ -1,23 +0,0 @@ -steps: - # The linter and all tests run inside this container. - # The venv image will be pulled if it doesn't exist locally. - - script: | - docker run \ - --tty \ - --detach \ - --name snekbox_test \ - --privileged \ - --hostname pdsnk-dev \ - -e PYTHONDONTWRITEBYTECODE=1 \ - -e PIPENV_PIPFILE="/snekbox/Pipfile" \ - -e ENV="${PWD}/scripts/.profile" \ - --volume "${PWD}":"${PWD}" \ - --workdir "${PWD}"\ - --entrypoint /bin/bash \ - pythondiscord/snekbox-venv:latest - displayName: 'Start Container' - - - script: | - docker exec snekbox_test /bin/bash -c \ - 'pipenv install --system --deploy --dev' - displayName: 'Install Development Dependencies' diff --git a/docker-compose.yml b/docker-compose.yml index aec17d3..066f38b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,7 +3,7 @@ services: snekbox: container_name: snekbox privileged: true - image: pythondiscord/snekbox:latest + image: ghcr.io/python-discord/snekbox:latest ports: - 8060:8060 init: true diff --git a/docker/Dockerfile b/docker/Dockerfile index b6c2f77..ea05c5c 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,4 +1,60 @@ -FROM pythondiscord/snekbox-venv:latest +FROM python:3.9-slim-buster as builder +RUN apt-get -y update \ + && apt-get install -y \ + bison=2:3.3.* \ + flex=2.6.* \ + g++=4:8.3.* \ + gcc=4:8.3.* \ + git=1:2.20.* \ + libprotobuf-dev=3.6.* \ + libnl-route-3-dev=3.4.* \ + make=4.2.* \ + pkg-config=0.29-6 \ + protobuf-compiler=3.6.* +RUN git clone \ + -b '2.9' \ + --single-branch \ + --depth 1 \ + https://github.com/google/nsjail.git /nsjail +WORKDIR /nsjail +RUN make + +FROM python:3.9-slim-buster as base +ENV PIP_NO_CACHE_DIR=false + +RUN apt-get -y update \ + && apt-get install -y \ + gcc=4:8.3.* \ + libnl-route-3-200=3.4.* \ + libprotobuf17=3.6.* \ + && rm -rf /var/lib/apt/lists/* +RUN pip install pipenv==2020.11.4 + +COPY --from=builder /nsjail/nsjail /usr/sbin/ +RUN chmod +x /usr/sbin/nsjail + +FROM base as venv +ARG DEV + +ENV PIP_NO_CACHE_DIR=false \ + PIPENV_DONT_USE_PYENV=1 \ + PIPENV_HIDE_EMOJIS=1 \ + PIPENV_NOSPIN=1 + +COPY Pipfile Pipfile.lock /snekbox/ +WORKDIR /snekbox + +RUN if [ -n "${DEV}" ]; \ + then \ + pipenv install --deploy --system --dev; \ + else \ + pipenv install --deploy --system; \ + fi + +# At the end to avoid re-installing dependencies when only a config changes. +COPY config/ /snekbox/config + +FROM venv ENTRYPOINT ["gunicorn"] CMD ["-c", "config/gunicorn.conf.py", "snekbox.api.app"] diff --git a/docker/base.Dockerfile b/docker/base.Dockerfile deleted file mode 100644 index a4a5ad1..0000000 --- a/docker/base.Dockerfile +++ /dev/null @@ -1,34 +0,0 @@ -FROM python:3.9-slim-buster as builder -RUN apt-get -y update \ - && apt-get install -y \ - bison=2:3.3.* \ - flex=2.6.* \ - g++=4:8.3.* \ - gcc=4:8.3.* \ - git=1:2.20.* \ - libprotobuf-dev=3.6.* \ - libnl-route-3-dev=3.4.* \ - make=4.2.* \ - pkg-config=0.29-6 \ - protobuf-compiler=3.6.* -RUN git clone \ - -b '2.9' \ - --single-branch \ - --depth 1 \ - https://github.com/google/nsjail.git /nsjail -WORKDIR /nsjail -RUN make - -FROM python:3.9-slim-buster -ENV PIP_NO_CACHE_DIR=false - -RUN apt-get -y update \ - && apt-get install -y \ - gcc=4:8.3.* \ - libnl-route-3-200=3.4.* \ - libprotobuf17=3.6.* \ - && rm -rf /var/lib/apt/lists/* -RUN pip install pipenv==2020.11.4 - -COPY --from=builder /nsjail/nsjail /usr/sbin/ -RUN chmod +x /usr/sbin/nsjail diff --git a/docker/venv.Dockerfile b/docker/venv.Dockerfile deleted file mode 100644 index 5c0fcfc..0000000 --- a/docker/venv.Dockerfile +++ /dev/null @@ -1,20 +0,0 @@ -FROM pythondiscord/snekbox-base:latest - -ARG DEV -ENV PIP_NO_CACHE_DIR=false \ - PIPENV_DONT_USE_PYENV=1 \ - PIPENV_HIDE_EMOJIS=1 \ - PIPENV_NOSPIN=1 - -COPY Pipfile Pipfile.lock /snekbox/ -WORKDIR /snekbox - -RUN if [ -n "${DEV}" ]; \ - then \ - pipenv install --deploy --system --dev; \ - else \ - pipenv install --deploy --system; \ - fi - -# At the end to avoid re-installing dependencies when only a config changes. -COPY config/ /snekbox/config diff --git a/scripts/check_dockerfiles.sh b/scripts/check_dockerfiles.sh deleted file mode 100755 index 88cb7cc..0000000 --- a/scripts/check_dockerfiles.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail -shopt -s inherit_errexit -exec 3>&1 # New file descriptor to stdout - -BASE_URL="https://dev.azure.com/\ -python-discord/${SYSTEM_TEAMPROJECTID}/_apis/build/builds?\ -queryOrder=finishTimeDescending&\ -resultFilter=succeeded&\ -\$top=1&\ -repositoryType=${BUILD_REPOSITORY_PROVIDER}&\ -repositoryId=${BUILD_REPOSITORY_NAME}&\ -api-version=5.0" - -declare -A build_cache - -get_build() { - local branch="${1:?"get_build: argument 1 'branch' is unset"}" - - # Attempt to use cached value - if [[ -v build_cache["${branch}"] ]]; then - printf '%s\n' "Retrieving build for ${branch} from cache." >&3 - printf '%s' "${build_cache[$branch]}" - return 0 - fi - - local url="${BASE_URL}&branchName=${branch}" - - printf '%s\n' "Retrieving the latest successful build using ${url}" >&3 - - local response - response="$(curl -sSL "${url}")" - - if [[ -z "${response}" ]] \ - || ! count="$(printf '%s' "${response}" | jq -re '.count')" \ - || (( "${count}" < 1 )) - then - return 1 - else - # Cache the response - build_cache["${branch}"]="${response}" - printf '%s' "${response}" - fi -} - -can_pull() { - local image="${1:?"can_pull: argument 1 'image' is unset"}" - - local master_commit - if master_commit="$( - get_build "refs/heads/master" \ - | jq -re '.value[0].sourceVersion' - )" \ - && git diff --quiet "${master_commit}" -- "${@:2}" - then - printf \ - '%s\n' \ - "Can pull ${image} image from Docker Hub; no changes since master." - - printf '%s\n' "##vso[task.setvariable variable=${image^^}_PULL]True" - else - printf \ - '%s\n' \ - "Cannot pull ${image} image from Docker Hub due to detected " \ - "changes; the ${image} image will be built." - - return 1 - fi -} - -# Get the previous commit -if [[ "${BUILD_REASON}" = "PullRequest" ]]; then - if ! prev_commit="$( - get_build "${BUILD_SOURCEBRANCH}" \ - | jq -re '.value[0].triggerInfo."pr.sourceSha"' - )" - then - echo \ - "Could not retrieve the previous build's commit." \ - "Falling back to the head of the target branch." - - prev_commit="origin/${SYSTEM_PULLREQUEST_TARGETBRANCH}" - fi -elif ! prev_commit="$( - get_build "${BUILD_SOURCEBRANCH}" \ - | jq -re '.value[0].sourceVersion' - )" -then - echo \ - "No previous build was found." \ - "Either the previous build is too old and was deleted" \ - "or the branch was empty before this build." \ - "All images will be built." - exit 0 -fi - -# Compare diffs -head="$(git rev-parse HEAD)" -printf '%s\n' "Comparing HEAD (${head}) against ${prev_commit}." - -if git diff --quiet "${prev_commit}" -- docker/base.Dockerfile; then - echo "No changes detected in docker/base.Dockerfile." - echo "##vso[task.setvariable variable=BASE_CHANGED]False" -else - # Always rebuild the venv if the base changes. - echo "Changes detected in docker/base.Dockerfile; all images will be built." - exit 0 -fi - -if git diff --quiet "${prev_commit}" -- docker/venv.Dockerfile Pipfile*; then - echo "No changes detected in docker/venv.Dockerfile or the Pipfiles." - echo "##vso[task.setvariable variable=VENV_CHANGED]False" - - if ! can_pull venv docker/venv.Dockerfile Pipfile*; then - # Venv image can't be pulled so it needs to be built. - # Therefore, the base image is needed too. - can_pull base docker/base.Dockerfile || true - fi -else - echo \ - "Changes detected in docker/venv.Dockerfile or the Pipfiles;" \ - "the venv image will be built." - - # Though base image hasn't changed, it's still needed to build the venv. - can_pull base docker/base.Dockerfile || true -fi diff --git a/scripts/dev.sh b/scripts/dev.sh index 408ce2e..f9fc016 100755 --- a/scripts/dev.sh +++ b/scripts/dev.sh @@ -5,12 +5,13 @@ if [ "$1" = "--build" ]; then shift - printf "Building pythondiscord/snekbox-venv:dev..." + printf "Building ghcr.io/python-discord/snekbox-venv:dev..." docker build \ - -t pythondiscord/snekbox-venv:dev \ - -f docker/venv.Dockerfile \ + -t ghcr.io/python-discord/snekbox-venv:dev \ + -f docker/Dockerfile \ --build-arg DEV=1 \ + --target venv \ -q \ . \ >/dev/null \ @@ -46,7 +47,7 @@ docker run \ --volume "${PWD}":"${PWD}" \ --workdir "${PWD}"\ --entrypoint /bin/bash \ - pythondiscord/snekbox-venv:dev \ + ghcr.io/python-discord/snekbox-venv:dev \ >/dev/null \ # Execute the given command(s) |