From 1e3005083bca22c04f890daca5cf7e74401b0aa3 Mon Sep 17 00:00:00 2001 From: Johannes Christ Date: Sat, 6 Jul 2024 13:13:44 +0200 Subject: Import all content --- content/_index.md | 6 + content/post/_index.md | 3 + ...rom-a-community-event-organizers-perspective.md | 102 +++++++++++++++ .../cover.png | Bin 0 -> 297560 bytes content/post/pixels-summer-2021.md | 102 +++++++++++++++ content/post/pixels-summer-2021/active-hours.png | Bin 0 -> 53947 bytes content/post/pixels-summer-2021/cover.png | Bin 0 -> 21917 bytes content/post/pixels-summer-2021/final-state.png | Bin 0 -> 159840 bytes .../pixels-summer-2021/most-popular-colors.png | Bin 0 -> 159565 bytes .../participation-statistics.svg | 1 + content/post/pixels-summer-2021/top-10-users.svg | 1 + .../statistics-infrastructure-at-python-discord.md | 138 +++++++++++++++++++++ .../cover.png | Bin 0 -> 210557 bytes .../grafana.png | Bin 0 -> 176962 bytes .../infraction-metrics.png | Bin 0 -> 134050 bytes .../postgresql-metrics.png | Bin 0 -> 136038 bytes .../redis-metrics.png | Bin 0 -> 162572 bytes 17 files changed, 353 insertions(+) create mode 100644 content/_index.md create mode 100644 content/post/_index.md create mode 100644 content/post/advent-of-code-from-a-community-event-organizers-perspective.md create mode 100644 content/post/advent-of-code-from-a-community-event-organizers-perspective/cover.png create mode 100644 content/post/pixels-summer-2021.md create mode 100644 content/post/pixels-summer-2021/active-hours.png create mode 100644 content/post/pixels-summer-2021/cover.png create mode 100644 content/post/pixels-summer-2021/final-state.png create mode 100644 content/post/pixels-summer-2021/most-popular-colors.png create mode 100644 content/post/pixels-summer-2021/participation-statistics.svg create mode 100644 content/post/pixels-summer-2021/top-10-users.svg create mode 100644 content/post/statistics-infrastructure-at-python-discord.md create mode 100644 content/post/statistics-infrastructure-at-python-discord/cover.png create mode 100644 content/post/statistics-infrastructure-at-python-discord/grafana.png create mode 100644 content/post/statistics-infrastructure-at-python-discord/infraction-metrics.png create mode 100644 content/post/statistics-infrastructure-at-python-discord/postgresql-metrics.png create mode 100644 content/post/statistics-infrastructure-at-python-discord/redis-metrics.png (limited to 'content') diff --git a/content/_index.md b/content/_index.md new file mode 100644 index 0000000..d677551 --- /dev/null +++ b/content/_index.md @@ -0,0 +1,6 @@ +--- +title: Python Discord +description: We're a large community focussed around the Python programming language. +cascade: + featured_image: "/images/social_banner_blank.png" +--- diff --git a/content/post/_index.md b/content/post/_index.md new file mode 100644 index 0000000..9bff67d --- /dev/null +++ b/content/post/_index.md @@ -0,0 +1,3 @@ +--- +title: Blog +--- diff --git a/content/post/advent-of-code-from-a-community-event-organizers-perspective.md b/content/post/advent-of-code-from-a-community-event-organizers-perspective.md new file mode 100644 index 0000000..105c4d9 --- /dev/null +++ b/content/post/advent-of-code-from-a-community-event-organizers-perspective.md @@ -0,0 +1,102 @@ +--- +title: Advent of Code from a community event organizer's perspective +description: "A look into what went on under the hood to pull off the Python Discord celebration of Advent of Code 2020." +author: "Janine (Kutiekatj9)" +date: 2021-03-24 +featured_image: "/post/advent-of-code-from-a-community-event-organizers-perspective/cover.png" +--- + + +## What is Advent of Code? + +Advent of Code is one of Python Discord's favorite events that happens each +year. An [Advent +Calendar](https://en.wikipedia.org/wiki/Advent_calendar?ref=blog.pythondiscord.com) +of small programming puzzles that are released each day from December 1st to +December 25th. The puzzles can be solved in any language, although we're +obviously partial to the python solutions. The problems are designed for a +variety of skill sets and skill levels, but it overall it gets harder as it +goes on. If you want to find out more about the event, you can take a look +here: https://adventofcode.com/about + +## How Python Discord participates + +All of us at Python Discord love Advent of Code. It's a great event that gets +people to learn, improve, and/or flex their coding skills with a community +event. We see a lot of people learn about new ways to accomplish fun and +interesting puzzles. Regex? Vim shortcuts and macros? Brute forcing with +powershell? All different solutions our server members have come up with. + +Part of how we engage our server is by having a community leaderboard that can +be viewed in server. It pulls the data from the Advent of Code leaderboard +itself, creates a fun and engaging environment. + + +## The Curse and Blessing of Server Growth + +I stepped into the Event Lead role 3 weeks before December 1st. Not a whole lot +of time, but we've run the event before so the prep shouldn't be too bad. I +spent the first week just getting acquainted with how the previous Advent of +Code was run and reading through any of the feedback we received. I also spent +some time getting familiar with Advent of Code itself! This was the first time +I had ever heard of it, nevermind participated. + +Fun fact about Advent of Code: an individual leaderboard has a limit of 200 +people. I was curious about how many people we had the previous year. After +some digging in archived channels and pestering our admins, I found out that +the 2019 Python Discord leaderboard had 177 people on it. The server in +December 2019 had 30,000 people in it. At the time of planning the 2020 AoC +event, we had over 100,000 members. + +Time for some back-of-the-envelope math! 2019 had 177 participants with 30,000 +people. Let's assume the % participation rate (0.0059 participants/server +member) stays the same year to year. From that ratio and applying it with our +current server size, we should expect roughly 590 people who want to join the +Python Discord leaderboard for the 2020 event. Ah, here lies the rub: +leaderboards are capped at 200 people. + +## Plan A + +I contacted the wonderful Advent of Code organizers to see if we could get an +increase for our leadboard. I asked (2 weeks away from the event) if they could +increase it and unfortunately it wasn't technically feasible at that time. + +... Time for Plan B! + +## Plan B + +It's clear we need multiple leaderboards. One thing the admin team and I agreed +on was that we still wanted one combined leaderboard for the server as a whole. +Having a cohesive and non-fragmented community is important to us, especially +for events that we encourage beginners to join. But we can't get really not +have multiple leaderboards, so the next step was to hack together the +leaderboards into one combined one. We knew we needed to update `@Sir Lancebot` +to grab the information from all the the various leaderboards we have (which +all have different session cookies that are required to access information +through the API), combine it into one leaderboard, and then *re-score the +entire leaderboard*. This has to happen everytime we need an update to the +leaderboard. (AoC organizers, if you're reading this don't worry! We had a +cooldown for when we would actually query the API.) + +Also, we realized we had to do all this a week before the event: code it, test +it, review it, and deploy it. Not a great start for the first event I was +officially leading. + +One of our wonderful owners Sebastiaan stayed up late several nights to get the +code working with caches, cooldowns, re-scoring logic, and the actual +displaying of the leaderboard. December 1st, midnight Eastern US time, we saw +our leaderboard go live! Overall the event was a great success. 447 people in +the server completed at least one day and a substantial amount completed all +the puzzles as well. I don't think we would've seen such great engagement +throughout the event without the combined leaderboard. A lot of the +1-hour-after-puzzle-release conversation was comparing positions on the +leaderboard, solutions, and talking about different approaches to the puzzle. + + +## Takeaways + +- You can never start planning too early! +- Advent of Code is a great experience and a great event to help strengthen + community. Definitely try it out in the future if you haven't. +- Sebastiaan is a fantastic human being who deserves cookies~ +- Server growth is awesome, but you've got to be prepared for what that brings. diff --git a/content/post/advent-of-code-from-a-community-event-organizers-perspective/cover.png b/content/post/advent-of-code-from-a-community-event-organizers-perspective/cover.png new file mode 100644 index 0000000..b7fbcf7 Binary files /dev/null and b/content/post/advent-of-code-from-a-community-event-organizers-perspective/cover.png differ diff --git a/content/post/pixels-summer-2021.md b/content/post/pixels-summer-2021.md new file mode 100644 index 0000000..bbb6a05 --- /dev/null +++ b/content/post/pixels-summer-2021.md @@ -0,0 +1,102 @@ +--- +title: "Python Discord Pixels: Summer 2021" +description: "Python Discord Pixels was our collaborative canvas event that we held between 25th May to the 14th June 2021 providing a beginner-friendly API to paint pixels on a virtual canvas." +author: Joe Banks, Chris Lovering +date: 2021-07-18 +featured_image: /post/pixels-summer-2021/cover.png +--- + +## What was Pixels? + +Python Discord Pixels was our collaborative canvas event that we held between +25th May to the 14th June 2021 providing a beginner-friendly API to paint +pixels on a virtual canvas. + +We built an API with FastAPI, Redis and PostgreSQL that would allow users to +paint a single pixel on a virtual canvas, with rate limits so they could only +do so a few times a minute. + +Painted pixels would then be returned to the API as well as posted every minute +into a channel in our Discord server so folks could keep up with the current +state of the canvas. + +The final state of the Pixels canvas: +![](./final-state.png) + +## Project motives + +The main goal for this project was for it to be a learning tool for those that +may not have ever interacted with an API before. By adding in a fun twist, +where users could see their actions reflected in an image displayed on our +server, we found users of all backgrounds participated. + +By giving participants a space to discuss implementation within our server, we +found that many of the more experienced users mentored those who were having +difficulties getting started. + +Many times throughout the event users were overjoyed when their first ever API +project finally worked, and they could see their first pixel on our canvas! + +## Participation statistics + +Across the 20 days that the event ran we saw 2,395,667 total pixel placements +from 414 distinct users. + +![](./participation-statistics.svg) + +## A look at the pixels + +You can see a timelapse [on our YouTube channel](https://www.youtube.com/watch?v=obC-l9JWx2M). + +## Colours + +Of the 2,395,667 placed pixels, there were 119,992 unique colours used, the +most popular being shown below: + +![](./most-popular-colors.png) + +## Active Hours + +The most active hour for placing pixels was 16:00 UTC with 115,312 pixels being +placed between 16:00–17:00 across the event. + +![](./active-hours.png) + +## Users + +The top 10 users by pixel placement are as follows: + +![](./top-10-users.svg) + +## Pixels Data + +We've made available the whole pixels dataset, with the majority of placed +pixels (pixels from banned users were not included). + +You can download the data at [this +link](https://cdn.discordapp.com/attachments/563594791770914816/858706547110838292/history.csv.gz?ref=blog.pythondiscord.com) +or [view it on +Kaggle](https://www.kaggle.com/joebanks/python-discord-pixels?ref=blog.pythondiscord.com). + + +## Source code + +We open sourced Pixels during the event so people could understand how the API +worked, we'll be writing a subsequent blog post to talk about the engineering +behind Pixels, and how we use tools such as PostgreSQL, Redis and more to +ensure reads and writes are as fast as they can be. + +You can find some of the documentation we used during the event +[here](https://pythondiscord.notion.site/Python-Discord-Pixels-99e3058855a94f6cab69853d6e2c355b?ref=blog.pythondiscord.com). +The repository can be found at +[python-discord/pixels](https://github.com/python-discord/pixels/?ref=blog.pythondiscord.com). + +## Future of Pixels + +Python Discord Pixels was a huge success, and outperformed our expectations in +terms of user participation. Due to the success we plan to bring back Pixels in +the future with different twists. Make sure to stay focused on our +#announcements channel in +[Discord](https://discord.gg/python?ref=blog.pythondiscord.com) and [our +Twitter](https://twitter.com/PythonDiscord?ref=blog.pythondiscord.com) to stay +up to date with future event announcements. diff --git a/content/post/pixels-summer-2021/active-hours.png b/content/post/pixels-summer-2021/active-hours.png new file mode 100644 index 0000000..c294e9b Binary files /dev/null and b/content/post/pixels-summer-2021/active-hours.png differ diff --git a/content/post/pixels-summer-2021/cover.png b/content/post/pixels-summer-2021/cover.png new file mode 100644 index 0000000..728db60 Binary files /dev/null and b/content/post/pixels-summer-2021/cover.png differ diff --git a/content/post/pixels-summer-2021/final-state.png b/content/post/pixels-summer-2021/final-state.png new file mode 100644 index 0000000..4074c69 Binary files /dev/null and b/content/post/pixels-summer-2021/final-state.png differ diff --git a/content/post/pixels-summer-2021/most-popular-colors.png b/content/post/pixels-summer-2021/most-popular-colors.png new file mode 100644 index 0000000..b197b19 Binary files /dev/null and b/content/post/pixels-summer-2021/most-popular-colors.png differ diff --git a/content/post/pixels-summer-2021/participation-statistics.svg b/content/post/pixels-summer-2021/participation-statistics.svg new file mode 100644 index 0000000..c04fd85 --- /dev/null +++ b/content/post/pixels-summer-2021/participation-statistics.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/content/post/pixels-summer-2021/top-10-users.svg b/content/post/pixels-summer-2021/top-10-users.svg new file mode 100644 index 0000000..9237975 --- /dev/null +++ b/content/post/pixels-summer-2021/top-10-users.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/content/post/statistics-infrastructure-at-python-discord.md b/content/post/statistics-infrastructure-at-python-discord.md new file mode 100644 index 0000000..fa451b1 --- /dev/null +++ b/content/post/statistics-infrastructure-at-python-discord.md @@ -0,0 +1,138 @@ +--- +title: Statistics infrastructure at Python Discord +description: "We have several tools setup to assist with the collection of statistics within Python Discord. We use these statistics to audit several things within the guild, such as help channel capacity, errors within our services and redundant channels." +author: "Joe Banks" +date: 2021-03-27 +featured_image: "/post/statistics-infrastructure-at-python-discord/cover.png" +--- + +We have several tools setup to assist with the collection of statistics within +Python Discord. We use these statistics to audit several things within the +guild, such as help channel capacity, errors within our services and redundant +channels. + +This article talks about several of the tools we use and problems we overcame +while building this system. + +## Grafana + +[Grafana](https://grafana.com/?ref=blog.pythondiscord.com) is the main +visualisation tool we use, it queries several data stores including PostgreSQL, +Prometheus and Graphite to render sweet looking graphs to our team. + +Some graphs that are displayed on our Kubernetes dashboard: +![](./grafana.png) + +## Graphite + +[Graphite](https://github.com/graphite-project?ref=blog.pythondiscord.com) is +our statistics ingress, it receives statistics from projects over the network +and writes them to the disk. It also provides a HTTP API that Grafana queries +to render metrics. + +## statsd + +[statsd](https://github.com/statsd/statsd?ref=blog.pythondiscord.com) is a +simple daemon for statistics accumulation providing a pleasant UDP API for +writing statistics. It's good for fire-and-forget metrics with minimal +overhead. + +In statsd, a metric packet looks like this: + +```sh +foo:1|c +``` + +This packet tells statsd (which eventually forwards to Graphite) to accumulate +the counter (indicated by the `c`) named `foo` by `1`. + +StatsD is just the method of accumulation before it forwards on to Graphite +which handles all storage and allows Grafana to query the statistics. We found +a docker image by the Graphite team called +[graphite-statsd](https://hub.docker.com/r/graphiteapp/graphite-statsd/?ref=blog.pythondiscord.com) +which allowed us to spin up an instance of Graphite and statsd together +allowing for writing & querying to the same service, dramatically simplifying +setup. + +## Python client + +Our main use for Graphite is accumulation of statistics from our Discord bot. +To do this we use the [`statsd` library for +Python](https://github.com/jsocol/pystatsd?ref=blog.pythondiscord.com). + +This is where we hit our first problem with the statistics infrastructure. +While our bot is written using asyncio, this client is not. To solve this we +had to write an asyncio UDP transport to deliver metrics to statsd without +risking our client blocking up the whole bot. + +This was a fairly easy solve though, all it required was a ~40 line +implementation of a async transport subclassing some base classes from the +`statsd` library. That code can be found +[here](https://github.com/python-discord/bot-core/blob/main/pydis_core/async_stats.py?ref=blog.pythondiscord.com). + +Since we run services in Kubernetes hooking up a client from any app is very +easy, all they have to do is connect to the service running on +`graphite.default.svc.cluster.local:8125` and start sending stats over UDP, +they'll immediately be queryable from Grafana. + +From a Python perspective, sending a new metric is as simple as: + +```python +self.bot.stats.incr("my_counter") +``` + +For Python Discord, the push architecture of Graphite suited us better than the +pull architecture of Prometheus since we did not want to have to open ports on +our bot container and write pull configurations every time we hooked up new +services. + +## Prometheus + +While the pull nature of +[Prometheus](https://prometheus.io/?ref=blog.pythondiscord.com) did not suit us +for Python Discord services such as the Discord bot or the site, it did suit +the monitoring of our critical services. + +We use several exporters for Prometheus to collect stats on different services: + +- [google/cadvisor](https://github.com/google/cadvisor?ref=blog.pythondiscord.com) + +- [prometheus-community/postgres-exporter](https://github.com/prometheus-community/postgres_exporter?ref=blog.pythondiscord.com) + +- [oliver006/redis_exporter](https://github.com/oliver006/redis_exporter?ref=blog.pythondiscord.com) + +- [prometheus/node_exporter](https://github.com/prometheus/node_exporter?ref=blog.pythondiscord.com) + +Combined, these exporters give us a great insight into how our core services +such as PostgreSQL and Redis are performing, and allow us to pick up any +performance snags long before they cause any major troubles. + +Our Redis metrics dashboard: + +![](./redis-metrics.png) + +Our PostgreSQL metrics dashboard: + +![](./postgresql-metrics.png) + +## PostgreSQL + +Of course, generating graphs of our non-statistics data is also incredibly +useful. We give Grafana read-access to certain tables in our primary +[PostgreSQL](https://www.postgresql.org/?ref=blog.pythondiscord.com) instance +so we can look at graphs of things like infraction distribution. + +A graph of infractions issues on our Discord server: + +![](./infraction-metrics.png) + +## Wrapping it up + +With all the above systems we've had great success increasing the observability +of our services, with Discord statistics proving useful to make moderation +decisions and system statistics helping to diagnose strange outages. + +The infrastructure we have selected works well for us, but there are definitely +improvements to it. If you have questions or suggestions we'd love to hear +them, feel free to ping `joe#6000` on Discord with any questions or +suggestions! diff --git a/content/post/statistics-infrastructure-at-python-discord/cover.png b/content/post/statistics-infrastructure-at-python-discord/cover.png new file mode 100644 index 0000000..719b33e Binary files /dev/null and b/content/post/statistics-infrastructure-at-python-discord/cover.png differ diff --git a/content/post/statistics-infrastructure-at-python-discord/grafana.png b/content/post/statistics-infrastructure-at-python-discord/grafana.png new file mode 100644 index 0000000..e648e7e Binary files /dev/null and b/content/post/statistics-infrastructure-at-python-discord/grafana.png differ diff --git a/content/post/statistics-infrastructure-at-python-discord/infraction-metrics.png b/content/post/statistics-infrastructure-at-python-discord/infraction-metrics.png new file mode 100644 index 0000000..9ec4aa1 Binary files /dev/null and b/content/post/statistics-infrastructure-at-python-discord/infraction-metrics.png differ diff --git a/content/post/statistics-infrastructure-at-python-discord/postgresql-metrics.png b/content/post/statistics-infrastructure-at-python-discord/postgresql-metrics.png new file mode 100644 index 0000000..dc011aa Binary files /dev/null and b/content/post/statistics-infrastructure-at-python-discord/postgresql-metrics.png differ diff --git a/content/post/statistics-infrastructure-at-python-discord/redis-metrics.png b/content/post/statistics-infrastructure-at-python-discord/redis-metrics.png new file mode 100644 index 0000000..79d35a4 Binary files /dev/null and b/content/post/statistics-infrastructure-at-python-discord/redis-metrics.png differ -- cgit v1.2.3