Deploying python pacakges to PyPi using gitlab and twine

While working on the web-minify project, I had to figure out just how it is possible to deploy a Python package to PyPi from gitlab. Here are my findings, hopefully it is useful for someone else!

Key information

  • The gitlab docker executor will run each job in a separate container.
  • No data is shared between jobs by default, so you have to use build artifacts to facilitate sharing between jobs
  • The build job can thus prepare the package data in the dist/ folder ready for deployment and then mark the dist/ folder as a build artifact
  • The deploy job will then have the dist/ folder available (all subsequent jobs after a build artifact has been defined will have access to the artifacts)
  • The deploy jog can then invoke twine to upload the package to PyPi
  • Twine takes a username and a password from a configuration file
  •  To avoid storing the password in the source code, a TWINE_PASSWORD environment variable is set in gitlab configuration of the project.
  • PyPi supports uploading packages using API tokens instead of requiring a username/password. In this mode the username is set to "__token__" and the password is actually a long token instead of a password. This allows us to have some granularity of permissions, creating a token that can only access one project instead of all the projects of a user.

You can look at the .gitlab-ci.yaml and Makefile of the  the web-minify project to see how this is done.


Docker registry credentials as secretGeneratow with kustomize

So you just discovered your new friend kustomize, and now you want to convert all your pesky secrets into secretGenerators. Good for you!

But... what about the docker registry credentials secret? It seems like magic that just works. Fear not, here is the definitive guide to how you can convert it to secretGenerator!

So let's assume you start with the following working setup:

Inside secret-docker-reg.yaml we have the following YAML:

apiVersion: v1
  - secret-docker-reg.json
kind: Secret
  name: docker-reg
  namespace: whatever
type: kubernetes.io/dockerconfigjson

Inside secret-docker-reg.json we have the following JSON:

{"auths":{"https://registry.gitlab.com":{"username":"yourusername","password":"SOME SECRET PASSWORD STRING","auth":"SOME SECRET AUTH STRING"}}}

Alternatively your secret-docker-reg.yaml may look like this:
apiVersion: v1
  - inlined-json: eyJhdXRocyI6eyJodHRwczovL3JlZ2lzdHJ5LmdpdGxhYi5jb20iOnsidXNlcm5hbWUiOiJ5b3VydXNlcm5hbWUiLCJwYXNzd29yZCI6IlNPTUUgU0VDUkVUIFBBU1NXT1JEIFNUUklORyIsImF1dGgiOiJTT01FIFNFQ1JFVCBBVVRIIFNUUklORyJ9fX0=
kind: Secret
  name: docker-reg
  namespace: whatever
type: kubernetes.io/dockerconfigjson

As you can see the difference is that in the first example the credentials is in a separate .json file referenced by the YAML (secret-docker-reg.json) while in the second the json content has been base64 encoded diretly into the YAML file.

Now to convert this to a kustomize secretGenerator you will need to keep the json in a separate file. In the first example you are already good to go, in the second simply copy the rather long base64 string (eyJhdXRocyI6eyJodHRwczovL3JlZ2lzdHJ5LmdpdGxhYi5jb20iOnsidXNlcm5hbWUiOiJ5b3VydXNlcm5hbWUiLCJwYXNzd29yZCI6IlNPTUUgU0VDUkVUIFBBU1NXT1JEIFNUUklORyIsImF1dGgiOiJTT01FIFNFQ1JFVCBBVVRIIFNUUklORyJ9fX0=)  into a separate file such as temp.txt and run the following command:
cat temp.txt | base64 -d > secret-docker-reg.json

This will decode the base64 into the json format in the file we want. NOTE: You could also use an online base64 conversion tool, but since this string contains credentials to your docker registry, you may want to use one you really trust and only over https.

What remains is to create the kustomize secretGenerator stanza:

- name: docker-reg
  - secret-docker-reg.json
  type: kubernetes.io/dockerconfigjson

Things to watch out for:

  • The secretGenerator stanza must be inside a kustomization.yaml file as it is part of kustomize and NOT part of kubernetes/kubectl.
  • The .json file needs to be on the same level or below the kustomize.yaml file from which it is referenced. If not you will get nasty errors about this.
And that's it! I hope this saved your ass like it would have saved mine if I had found it before I knew. Oh wait...


Happy Anniversary 2020!

The 7th of January 2020 marks the four year anniversary of the OctoMY™ project. The project has had some amazing progress as always, and just a few months ago a new strategy was formed that may slow development down momentarily.

More on that later.

Suffice to say, I am happy as ever to work on this project, and I am looking forward to an amazing 2020.