diff options
Diffstat (limited to 'docs/meeting_notes')
23 files changed, 951 insertions, 0 deletions
diff --git a/docs/meeting_notes/2022-04-07.rst b/docs/meeting_notes/2022-04-07.rst new file mode 100644 index 0000000..21642d3 --- /dev/null +++ b/docs/meeting_notes/2022-04-07.rst @@ -0,0 +1,20 @@ +DevOps Meeting Notes +==================== + +Agenda +------ + +- No updates, as last week’s meeting did not take place + +Roadmap review & planning +------------------------- + +What are we working on for the next meeting? + +- Help wanted for #57 (h-asgi) +- #58 (postgres exporter) needs a new review +- #54 (firewall in VPN) will be done by Johannes +- We need a testing environment #67 +- Johannes will add a Graphite role #31 +- Sofi will take a look at #29 +- #41 (policy bot) will be taken care of by Johannes diff --git a/docs/meeting_notes/2022-09-18.rst b/docs/meeting_notes/2022-09-18.rst new file mode 100644 index 0000000..f6b56c2 --- /dev/null +++ b/docs/meeting_notes/2022-09-18.rst @@ -0,0 +1,74 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- Joe will grant Chris access to the netcup hosts. + +NetKube status +~~~~~~~~~~~~~~ + +- **Rollout** + + - ☒ RBAC configuration and access granting + - ☒ Most nodes are enrolled, Joe will re-check + - ``turing``, ``ritchie``, ``lovelace`` and ``neumann`` will be + Kubernetes nodes + - ``hopper`` will be the storage server + +- **Storage drivers** + + - Not needed, everything that needs persistent storage will run on + hopper + - Netcup does not support storage resize + - We can download more RAM if we need it + - A couple of services still need volume mounts: Ghost, Grafana & + Graphite + +- **Control plane high availability** + + - Joe mentions that in the case the control plane dies, everything + else will die as well + - If the control plane in Germany dies, so will Johannes + +- **Early plans for migration** + + - We can use the Ansible repository issues for a good schedule + - Hopper runs ``nginx`` + - Statement from Joe: > “There is an nginx ingress running on every + node in the cluster, okay, > okay? We don’t, the way that’s, + that’s as a service is a NodePort, right? > So it has a normal IP, + but the port will be like a random port in the range > of the + 30,000s. Remember that? Hold on. Is he writing rude nodes? And + then… > We have nginx, so this is where it’s like a little bit, + like, not nice, I > guess we just like, cronjob it, to pull the + nodes, like, every minute or > so, and then update the config if + they change. But then it’s just like… > nginx is like a catalogue + of nodes. Wahhh, you drive me crazy.” + + - “Nah, it makes sense!” + + - “It does!” + + - Joe will figure this out with assistance from his voices. + +Open authentication +~~~~~~~~~~~~~~~~~~~ + +- Joe and Johannes will check out OpenLDAP as a JumpCloud alternative + starting from this evening +- Sofi has experience with OpenLDAP + +Sponsorship +----------- + +This meeting has been sponsored by Chris Hemsworth Lovering’s +relationship therapy company, “Love To Love By Lovering”. You can sign +up by sending a mail to [email protected]. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2022-10-05.rst b/docs/meeting_notes/2022-10-05.rst new file mode 100644 index 0000000..c405e01 --- /dev/null +++ b/docs/meeting_notes/2022-10-05.rst @@ -0,0 +1,13 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- Joe Banks configured proper RBAC for Chris, Johannes and Joe himself + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2022-10-19.rst b/docs/meeting_notes/2022-10-19.rst new file mode 100644 index 0000000..fa51d32 --- /dev/null +++ b/docs/meeting_notes/2022-10-19.rst @@ -0,0 +1,31 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- One hour of gartic phone, for team spirit. +- Created user accounts for Sofi and Hassan +- Joe created an architecture diagram of the NGINX setup + + - *This is still in Notion* + +- Joe explained his NGINX plans: > “It’s not actually that hard, right? + So you spawn 5 instances of nginx in a > DaemonSet, because then one + gets deployed to every node okay, following? > Then we get NodePort, + instead of LoadBalancers or whatever, which will get > a random port + allocatead in the 35000 range, and that will go to nginx, and > on + each of those ports, it will go to nginx, right? And then we poll the + > Kubernetes API and what is the port that each of these nginx + instances is > running on, and add that into a roundrobin on the + fifth node. Right? Yeah. > That’s correct. That won’t do TLS though, + so that will just HAProxy. Yeah.” +- Joe will terminate our JumpCloud account +- Chris reset the Minecraft server +- Email alerting needs to be configured + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2022-10-26.rst b/docs/meeting_notes/2022-10-26.rst new file mode 100644 index 0000000..5684d7f --- /dev/null +++ b/docs/meeting_notes/2022-10-26.rst @@ -0,0 +1,18 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- Chris upgraded PostgreSQL to 15 in production +- Johannes added the Kubernetes user creation script into the + Kubernetes repository in the docs + +*(The rest of the meeting was discussion about the NetKube setup, which +has been scrapped since)*. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2022-11-02.rst b/docs/meeting_notes/2022-11-02.rst new file mode 100644 index 0000000..010b8f0 --- /dev/null +++ b/docs/meeting_notes/2022-11-02.rst @@ -0,0 +1,27 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +Hanging behaviour of ModMail +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- `Source <https://discord.com/channels/267624335836053506/675756741417369640/1036720683067134052>`__ + +- Maybe use `Signals + a + debugger <https://stackoverflow.com/a/25329467>`__? + +- … using `something like pdb for the + debugger <https://wiki.python.org/moin/PythonDebuggingTools>`__? + +- Or `GDB, as it seems handy to poke at stuck multi-threaded python + software <https://wiki.python.org/moin/DebuggingWithGdb>`__? + +- ModMail has been upgraded to version 4 + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2022-11-23.rst b/docs/meeting_notes/2022-11-23.rst new file mode 100644 index 0000000..5f74fc6 --- /dev/null +++ b/docs/meeting_notes/2022-11-23.rst @@ -0,0 +1,30 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +*(This meeting was mostly about NetKube, with the following strange text +included, and everything outside of the text has been removed since the +NetKube plans have been scrapped)*. + +Joe Banks, after a month-long hiatus to become a dad to every second +girl on uni campus, has managed to pull up to the DevOps meeting. + +We are considering using Kubespray (https://kubespray.io/#/) in order to +deploy a production-ready bare-metal Kubernetes cluster without +involvement from Joe “Busy With Poly Girlfriend #20” Banks. + +At the moment cluster networking is not working and Joe mentions that +the last time he has touched it, it worked perfectly fine. However, the +last time he touched it there was only 1 node, and therefore no +inter-node communications. + +Joe thinks he remembers installing 3 nodes, however, we at the DevOps +team believe this to be a marijuana dream + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-02-08.rst b/docs/meeting_notes/2023-02-08.rst new file mode 100644 index 0000000..c65193c --- /dev/null +++ b/docs/meeting_notes/2023-02-08.rst @@ -0,0 +1,17 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- Investigation into deploying a VPN tool such as WireGuard to have + inter-node communication between the Netcup hosts. + +*(The rest of this meeting was mostly about NetKube, which has since +been scrapped)*. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-02-21.rst b/docs/meeting_notes/2023-02-21.rst new file mode 100644 index 0000000..c30c133 --- /dev/null +++ b/docs/meeting_notes/2023-02-21.rst @@ -0,0 +1,31 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +Reusable status embed workflows +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Further discussion with Bella followed +- Upstream pull request can be found at + `python-discord/bot#2400 <https://github.com/python-discord/bot/pull/2400>`__ + +Local vagrant testing setup +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Our new `testing setup using Vagrant + VMs <https://github.com/python-discord/infra/pull/78>`__ has been + merged. + +A visit from Mina +~~~~~~~~~~~~~~~~~ + +Mina checked in to make sure we’re operating at peak Volkswagen-like +efficiency. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-02-28.rst b/docs/meeting_notes/2023-02-28.rst new file mode 100644 index 0000000..fe7dc47 --- /dev/null +++ b/docs/meeting_notes/2023-02-28.rst @@ -0,0 +1,16 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- Black knight’s CI & dependabot configuration has been mirrored across + all important repositories + +- The test server has been updated for the new configuration + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-05-16.rst b/docs/meeting_notes/2023-05-16.rst new file mode 100644 index 0000000..bafa941 --- /dev/null +++ b/docs/meeting_notes/2023-05-16.rst @@ -0,0 +1,15 @@ +DevOps Meeting Notes +==================== + +*Migrated from Notion*. + +Agenda +------ + +- Bella set up `CI bot docker image + build <https://github.com/python-discord/bot/pull/2603>`__ to make + sure that wheels are available. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-07-11.rst b/docs/meeting_notes/2023-07-11.rst new file mode 100644 index 0000000..6c51f1c --- /dev/null +++ b/docs/meeting_notes/2023-07-11.rst @@ -0,0 +1,41 @@ +DevOps Meeting Notes +==================== + +Participants +------------ + +- Chris, Johannes, Bella, Bradley + +Agenda +------ + +New Ansible setup +~~~~~~~~~~~~~~~~~ + +Chris presented the new Ansible setup he’s been working on. We plan to +use WireGuard for networking. We agreed that selfhosting Kubernetes is +not the way to go. In general, the main benefit from switching away to +Linode to Netcup is going to be a ton more resources from the Netcup +root servers we were given. The original issue with Linode’s AKS of +constantly having problems with volumes has not been present for a +while. Chris mentions the one remaining issue is that we’re at half our +memory capacity just at idle. + +It’s our decision where to go from here - we can stick to the Kubernetes +setup or decide on migrating to the Ansible setup. But we have bare +metal access to the Netcup hosts, which makes e.g. managing databases a +lot easier. Chris mentions the possibility to only use Netcup for our +persistence and Linode AKS for anything else, but this has the issue of +us relying on two sponsors for our infrastructure instead of one. + +PostgreSQL was set up to run on ``lovelace``. + +Decision +~~~~~~~~ + +**It was decided to hold a vote on the core development channel, which +will be evaluated next week to see how to proceed with the setup**. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-07-18.rst b/docs/meeting_notes/2023-07-18.rst new file mode 100644 index 0000000..28f6c88 --- /dev/null +++ b/docs/meeting_notes/2023-07-18.rst @@ -0,0 +1,42 @@ +DevOps Meeting Notes +==================== + +Secret management improvements +------------------------------ + +To allow for **better management of our Kubernetes secrets**, Chris set +out to configure ``git-crypt`` in GPG key mode. For comparison, the +previous approach was that secrets were stored in Kubernetes only and +had to be accessed via ``kubectl``, and now ``git-crypt`` allows us to +transparently work with the files in unencrypted manner locally, whilst +having them secure on the remote, all via ``.gitattributes``. + +The following people currently have access to this: + +- Johannes Christ [email protected] + (``8C05D0E98B7914EDEBDCC8CC8E8E09282F2E17AF``) +- Chris Lovering [email protected] + (``1DA91E6CE87E3C1FCE32BC0CB6ED85CC5872D5E4``) +- Joe Banks [email protected] (``509CDFFC2D0783A33CF87D2B703EE21DE4D4D9C9``) + +For Hassan, we are still waiting on response regarding his GPG key +accuracy. + +The pull request for the work can be found `at +python-discord/kubernetes#156 <https://github.com/python-discord/kubernetes/pull/156>`__. + +**To have your key added, please contact any of the existing key +holders**. More documentation on this topic is pending to be written, +see +`python-discord/kubernetes#157 <https://github.com/python-discord/kubernetes/issues/157>`__. + +Infrastructure migration decision +--------------------------------- + +The voting started `last week <./2023-07-11.md>`__ will be properly +talked about `next week <./2023-07-25.md>`__, so far it looks like we’re +definitely not selfhosting Kubernetes at the very least. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-07-25.rst b/docs/meeting_notes/2023-07-25.rst new file mode 100644 index 0000000..be4d20c --- /dev/null +++ b/docs/meeting_notes/2023-07-25.rst @@ -0,0 +1,4 @@ +DevOps Meeting Notes +==================== + +Postponed to next week due to absence. diff --git a/docs/meeting_notes/2023-08-01.rst b/docs/meeting_notes/2023-08-01.rst new file mode 100644 index 0000000..925417a --- /dev/null +++ b/docs/meeting_notes/2023-08-01.rst @@ -0,0 +1,66 @@ +DevOps Meeting Notes +==================== + +Agenda +------ + +Infrastructure migration +~~~~~~~~~~~~~~~~~~~~~~~~ + +The vote is tied. Chris and Johannes decided that we should test out +migrating the PostgreSQL database at the very least. We then have more +freedom about our data. What we need to do: + +- Allow PostgreSQL connections from LKE’s static IPs in the firewall +- Whitelist the static IPs from Linode via ``pg_hba.conf`` +- Schedule downtime for the PostgreSQL database +- **At downtime** + + - Take writers offline + - Dump database from Linode into Netcup + - Update all the client’s database URLs to point to netcup + - Restart writers + +We want to rely on the restore to create everything properly, but will +need to test run this beforehand. The following ``pg_virtualenv`` +command has showcased that it works properly: + +.. code:: sh + + kubectl exec -it postgres-... -- pg_dumpall -U pythondiscord \ + | pg_virtualenv psql -v ON_ERROR_STOP=1 + +Note however that the database extension ``pg_repack`` needs to be +installed. + +Before we can get started, we need to allow the PostgreSQL role to +configure ``pg_hba.conf`` and ``postgresql.conf`` entries. + +Meeting notes +~~~~~~~~~~~~~ + +We’re using GitHub at the moment. Some are left in Notion. We should +migrate these to GitHub to have a uniform interface: Johannes will pick +up +`python-discord/infra#108 <https://github.com/python-discord/infra/issues/108>`__ +to merge them together into Git, as its more open than Notion. + +Ansible lint failures in the infra repository +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Excluding the vault was found as the working solution here, as +implemented by Chris. + +Kubernetes repository pull requests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These were cleaned up thanks to Chris. + +Roadmap review & planning +------------------------- + +- Chris will prepare the PostgreSQL configuration mentioned above. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-08-08.rst b/docs/meeting_notes/2023-08-08.rst new file mode 100644 index 0000000..4b06d5e --- /dev/null +++ b/docs/meeting_notes/2023-08-08.rst @@ -0,0 +1,54 @@ +DevOps Meeting Notes +==================== + +Agenda +------ + +- Configuration of PostgreSQL and the PostgreSQL exporter + + - **No time so far**. Chris has been busy with renovating his living + room, and Johannes has been busy with renovating his bedroom. + Bradley prefers to remain quiet. + + - Chris will try to work on this in the coming week and will try to + have Bella around as well, since he wanted to join the setup. + +- **Potential slot for GPG key signing of DevOps members**. External + verification will be necessary. + + - Skipped. No webcam on Chris. + +- We need to assign a **librarian** to keep our documents organized + according to a system. Johannes is happy to do this for now. + + - Let’s move the existing documentation from the Kubernetes + repository into the infra repository. See + `kubernetes#161 <https://github.com/python-discord/kubernetes/issues/161>`__. + + - **Our Notion DevOps space is full of junk**. Outside of that, it’s + not open to read for outside contributors, and does not leave much + choice over which client to use for editing content. + + - Chris agrees, without looking on it - just from memory. We + should move it to the infra repository. (The meeting notes have + already been transferred). + + - Bella suggests to add some automation to make keeping everything + in clean order less tedious. + +- We may want to integrate the **Kubernetes repository** and the infra + repository together altogether, however there are a lot of + repositories referencing the deployment manifests that would need to + be updated. + + - Chris mentions that regardless of what we do, we should - at the + very least move all documentation into the ``infra`` repository, + including the static site generator. At the moment we’re using + Jekyll but we’re open to trying alternatives such as Hugo. + +- We closed some issues and pull requests in the repositories for late + spring cleaning. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2 autoindent conceallevel=2: --> diff --git a/docs/meeting_notes/2023-08-22.rst b/docs/meeting_notes/2023-08-22.rst new file mode 100644 index 0000000..67f53e9 --- /dev/null +++ b/docs/meeting_notes/2023-08-22.rst @@ -0,0 +1,40 @@ +DevOps Meeting Notes +==================== + +.. raw:: html + + <!-- + + Useful links + + - Infra open issues: https://github.com/python-discord/infra/issues + + - infra open pull requests: https://github.com/python-discord/infra/pulls + + - *If* any open issue or pull request needs discussion, why was the existing + asynchronous logged communication over GitHub insufficient? + + --> + +Agenda +------ + +- Bella said he is on the streets. **We should start a gofundme**. + + - After some more conversation this just means he is on vacation and + currently taking a walk. + +- Chris has been busy with turning his living room into a picasso art + collection, Johannes has been busy with renovating his bedroom, and + Bella is not home. + + - Our next priority is winning. + +- We checked out some issues with documentation generation in + ``bot-core`` that Bella has mentioned. We managed to fix one issue + with pydantic by adding it to an exclude list but ran into another + problem next. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-08-29.rst b/docs/meeting_notes/2023-08-29.rst new file mode 100644 index 0000000..8e0a7d4 --- /dev/null +++ b/docs/meeting_notes/2023-08-29.rst @@ -0,0 +1,65 @@ +DevOps Meeting Notes +==================== + +.. raw:: html + + <!-- + + Useful links + + - Infra open issues: https://github.com/python-discord/infra/issues + + - infra open pull requests: https://github.com/python-discord/infra/pulls + + - *If* any open issue or pull request needs discussion, why was the existing + asynchronous logged communication over GitHub insufficient? + + --> + +Agenda +------ + +- **Bella is still on the streets** + + - The Python Discord Bella On The Streets Fundraising Campaign Q3 + 2023 has not been successful so far. To help Bella receive French + citizenship, Joe has put up a French flag behind himself in the + meeting. + + - Joe corrects my sarcasm. It is an Italian flag, not a French + flag. The reason for this flag is that his new prime interest + on campus was born in Italy. + +- **The SnekBox CI build is pretty slow** + + - Guix and Nix are not alternatives. Neither is Ubuntu + + - We use pyenv to build multiple Python versions for a new feature + + - The feature is not rolled out yet + + - Part of the problem is that we build twice in the ``build`` and + the ``deploy`` stage + + - On rollout, Joe tested it and it works fine + +- No update on the Hugo build yet + +- For snowflake, Johannes will write a proposal to the admins for + hosting it + + - We should consider talking about the following points: + + - statistically ~8% of Tor traffic is problematic (10% of traffic + is to hidden services, 80% of hidden service traffic is for + illegal services) + + - overall the project’s position and our ideal is to help people + for a good cause + + - all traffic is forwarded to the Tor network, the service is + lightweight and only proxies encrypted traffic there + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-09-05.rst b/docs/meeting_notes/2023-09-05.rst new file mode 100644 index 0000000..2c80c2e --- /dev/null +++ b/docs/meeting_notes/2023-09-05.rst @@ -0,0 +1,53 @@ +DevOps Meeting Notes +==================== + +.. raw:: html + + <!-- + + Useful links + + - Infra open issues: https://github.com/python-discord/infra/issues + + - infra open pull requests: https://github.com/python-discord/infra/pulls + + - *If* any open issue or pull request needs discussion, why was the existing + asynchronous logged communication over GitHub insufficient? + + --> + +Agenda +------ + +- No update on the Hugo build yet + +- Johannes wrote a proposal for snowflake proxy to be deployed to our + netcup hosts + + - Admins discussed and came to the conclusion that since we don’t + own the servers, we got the servers from netcup as a sponsorship + to host our infra, so using them to host something that isn’t our + infra doesn’t seem right. + +- Lots of dependabot PRs closed + + - https://github.com/search?q=org%3Apython-discord++is%3Apr+is%3Aopen+label%3A%22area%3A+dependencies%22&type=pullrequests&ref=advsearch + - Closed ~50% of PRs + +- Workers repo has had its CI rewritten, all workers have consistent + package.json, scripts, and using the new style of cloudflare workers + which don’t use webpack + +- Metricity updated to SQLAlchemy 2 + +- Olli CI PR is up + + - https://github.com/python-discord/olli/pull/25 + +- Sir-Robin pydantic constants PR is up + + - https://github.com/python-discord/sir-robin/pull/93 + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2023-09-12.rst b/docs/meeting_notes/2023-09-12.rst new file mode 100644 index 0000000..7bfcd1a --- /dev/null +++ b/docs/meeting_notes/2023-09-12.rst @@ -0,0 +1,73 @@ +DevOps Meeting Notes +==================== + +.. raw:: html + + <!-- + + Useful links + + - Infra open issues: https://github.com/python-discord/infra/issues + + - infra open pull requests: https://github.com/python-discord/infra/pulls + + - *If* any open issue or pull request needs discussion, why was the existing + asynchronous logged communication over GitHub insufficient? + + --> + +Agenda +------ + +- We have reason to believe that Bella is still on the streets. Worse, + Bella is not available at the moment, leading us to believe that + Bella has still not found a home. + + - Eight minutes into the meeting, Bella joins, complaining about the + bad internet. He mentions he is still on the streets (this may + contribute to the bad internet factor). + +- Chris made Mina leave with his repeated comments about Bella being + homeless, reminding Mina of the growing unemployment rate within the + DevOps team. As head of HR she cannot further support this matter. + +- About #139, Bella mentions that online websites may cover the same + need that we have, but it may not be really useful for having it as a + command. + + - Chris adds that “if someone wants to do it, I don’t mind” and “I + don’t think it would be very useful for a command, but I think it + would be fun to learn for someone implementing it”. As long as + whoever is implementing is is aware that it would not be used too + much, it would be fine. + +- No progress on the hugo front + +- Our email service with workers will be forward only + + - With postfix you will be able to reply. Joe wants to have an + excuse to play with Cloudflare workers though. + +- `50 open pull requests from + dependabot <https://github.com/search?q=org%3Apython-discord++is%3Apr+is%3Aopen+author%3Aapp%2Fdependabot&type=pullrequests&ref=advsearch>`__ + + - Tip from The Man: press ^D to make a bookmark in your browser + + - “Those can just be blindly merged” - Chris + +- Grouping of dependencies: Dependabot now allows you to group together + multiple dependency updates into a single pull request. + + - Possible approaches suggested: Group all the docker updates + together, group any linting dependencies together (would just + require a big RegEx). Dependabot natively works with its own + dependency groups here (e.g. Docker, Pip). + +- Mr. Hemlock wants to raise his roof: It’s his project for this + Autumn. We, the team, are looking forward to his project - especially + Bella, who is currently looking for housing. “It’s all coming + together”, said Chris to the situation. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/2024-07-02.rst b/docs/meeting_notes/2024-07-02.rst new file mode 100644 index 0000000..029d53e --- /dev/null +++ b/docs/meeting_notes/2024-07-02.rst @@ -0,0 +1,171 @@ +DevOps Meeting Notes +==================== + +.. raw:: html + + <!-- + + Useful links + + - Infra open issues: https://github.com/python-discord/infra/issues + + - infra open pull requests: https://github.com/python-discord/infra/pulls + + - *If* any open issue or pull request needs discussion, why was the existing + asynchronous logged communication over GitHub insufficient? + + --> + +Attendees +--------- + +Joe and Johannes. + +Chris unfortunately died in a fatal train accident and could not attend +the meeting. This incident will be rectified in the next release, +“Lovering 2.0: Immortability”. + +Bella is out on the streets again. We are waiting for approval from the +Python Discord admins to run another fundraiser. + +Agenda +------ + +- **Configuration of renovate** (Joe) + + We are replacing dependabot with renovatebot. Johannes welcomes this + decision. Joe says we are looking for automatic deployment from + Kubernetes to make sure that any updates are automatically deployed. + + **Conclusion**: Implemented. + +- **Resizing Netcup servers** (Joe, Johannes) + + We can probably get rid of turing, assess what else we want to deploy + on lovelace, and then ask for a resize. + + **Conclusion**: Create issue to move things off turing, remove it + from the inventory, remove it from documentation, power it off, then + have Joe ask for server removal. + +- **Updating the public statistics page** (Johannes) + + Discussing and showcasing possible alternatives to the current + infrastructure powering https://stats.pythondiscord.com via the + https://github.com/python-discord/public-stats repository. Johannes + presents his current scripts that cuddle RRDTool into loading data + out of metricity, Joe says we will discuss with Chris what to do + here. + + The likely way going forward will be that *we will open an issue to + set it up*, the setup will contain an Ansible role to deploy the + cronjob and the script onto lovelace alongside with the ``rrdtool`` + PostgreSQL user. + + **Conclusion**: Johannes will create an issue and codify the setup in + Ansible. + +- **New blog powered by Hugo** (Johannes) + + Our current Ghost-powered blog is a tiny bit strange, and the + onboarding ramp to contribute articles is large. We want to migrate + this to Hugo - Johannes is leading the effort on it. The main work + will be building an appropriate theme, as no nicely suitable + replacement theme has been found so far. Front-end contributors would + be nice for this, although currently everything is still local on my + machine. + + Joe mentions that we don’t need to take anything particularly similar + to the current Ghost theme, just some vague resemblance would be + nice. Most of the recommended Hugo themes would probably work. + Johannes will check it out further. + + **Conclusion**: Try the `hugo-casper-two + theme <https://github.com/eueung/hugo-casper-two>`__ and report back. + +- **Finger server** (Joe, Johannes) + + Joe recently proposed `the deployment of a finger + server <https://github.com/python-discord/infra/pull/373>`__. Do we + want this and if yes, how are we going to proceed with this? If we do + not want any, running the ``pinky`` command locally or via ``ssh`` + would be a sound idea. We also need to consider whether members will + update their files regularly - we may want to incorporate + functionality for this into e.g. King Arthur. + + Joe says that we shouldn’t put a lot of development effort into it, + it would be simply a novelty thing. + + **Conclusion**: This is a nice cheap win for some fun which should + just be a simple Python file (via Twisted’s Finger protocol support + or whatever) that connects to LDAP (see Keycloak authentication + server) and outputs information. We could possibly integrate this + into King Arthur as well, so the querying workflow could look like KA + -> fingerd -> LDAP, or people could use finger commands directly. + +- **Keycloak authentication server** (Joe) + + Joe mentions that we are deploying a Keycloak server because for some + members authenticating via GitHub is cumbersome, for instance because + their GitHub account is connected to their employer’s GitHub + Enterprise installation. We could hook up a finger server to the LDAP + endpoint. Joe also mentions that we might want to set up e-mail + forwarding from pydis addresses to users via the user database that + will be stored in Keycloak. + + Currently we only have a Keycloak installation that stores items in + PostgreSQL. This installation can federate to LDAP - we would simply + have to settle on some directory service backend. Joe suggests + FreeIPA because he’s familar with it (including the Keycloak + integration). The problem is that it doesn’t work on Debian. The + alternative proposal, given that we’re saving ~50$/month on Linode, + would be spinning up a Rocky VM with FreeIPA on it on Linode (we + already have the budget) or ask Netcup for another VM. Ultimately, + the system to run FreeIPA would be something CentOS-based. One aspect + to consider is networking security: in Linode we could use their + private cloud endpoint feature to securely expose the LDAP server to + Keycloak and other services in Kubernetes, if we were to run it in + Netcup, we would need to use a similar setup to what we currently + have with PostgreSQL. + + Any Python Discord user would be managed in LDAP, and Keycloak has + the necessary roles to write back into LDAP. Keeping the users in + FreeIPA up-to-date would be a somewhat manual procedure. Joe’s plan + was to pick up the user’s Discord username and use + ``[email protected]`` as their name and do account setup as part of + the staff onboarding. + + **Conclusion**: Will wait for Chris to discuss this further, but we + simply need to decide where we want to run the LDAP service. + +- **Flux CD** (Joe) + + Joe proposes deploying `flux <https://fluxcd.io/>`__ as a way to + improve the way we manage our CI/CD. We want the cluster to be able + to synchronize its state with the git repository. There are some + manifests in the repository currently that are not in sync with the + cluster version. + + **Conclusion**: Approved, Joe will create an issue and do it. + +- **Polonium** (Chris) + + Question came up regarding why the bot does not write to the database + directly. Joe said it’s not perfect to have the bot write to it + directly - in metricity it works but it’s not perfect. Chris probably + had good reason: separation of intent. + + **Conclusion**: Approved, write to R&D for financing. + +- **Rethinking Bella: Suggested measures to gain autonomy** (Chris) + + Chris will present our current plans to biologically re-think and + improve Bella’s current architecture by means of + hypertrophy-supported capillary enlargements, with the final goal of + gaining complete control and ownership over the World Economic Forum + by 2026. As Bella is currently on parental leave, we will send him + the result of this voting via NNCP. + +.. raw:: html + + <!-- vim: set textwidth=80 sw=2 ts=2: --> diff --git a/docs/meeting_notes/index.rst b/docs/meeting_notes/index.rst new file mode 100644 index 0000000..cf7bb14 --- /dev/null +++ b/docs/meeting_notes/index.rst @@ -0,0 +1,31 @@ +Meeting notes +============= + +Minutes for previous Devops meetings. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + template + 2022-04-07 + 2022-09-18 + 2022-10-05 + 2022-10-19 + 2022-10-26 + 2022-11-02 + 2022-11-23 + 2023-02-08 + 2023-02-21 + 2023-02-28 + 2023-05-16 + 2023-07-11 + 2023-07-18 + 2023-07-25 + 2023-08-01 + 2023-08-08 + 2023-08-22 + 2023-08-29 + 2023-09-05 + 2023-09-12 + 2024-07-02 diff --git a/docs/meeting_notes/template.rst b/docs/meeting_notes/template.rst new file mode 100644 index 0000000..e8bc719 --- /dev/null +++ b/docs/meeting_notes/template.rst @@ -0,0 +1,19 @@ +DevOps Meeting Notes +==================== + +.. + Useful links + + - Infra Kanban board: https://github.com/orgs/python-discord/projects/17/views/4 + + - Infra open issues: https://github.com/python-discord/infra/issues + + - infra open pull requests: https://github.com/python-discord/infra/pulls + + - *If* any open issue or pull request needs discussion, why was the existing + asynchronous logged communication over GitHub insufficient? + +Agenda +------ + +.. vim: set textwidth=80 sw=2 ts=2: |