Hello all, it’s been a bit. Sorry about that.
This is aismallard, I’ll be speaking about my perspective and work on Wikijump. Where we are now, why that is, and where I’d like to go from here.
Past Missteps
First I think there are a number of items which, while obvious in hindsight, contributed to a lot of delays which could’ve been avoided.
This project originally began because bluesoul was able to, using a combination of publicly available resources and help from former contributors, piece together a Wikidot.org build. For those unaware, this was a (discarded) project to have an open source Wikidot that anyone could host themselves. This is the origin of the gabrys repository, an AGPL-licensed version of Wikidot which is about a decade out of date.
To explain, when I saw that it’s a decade out of date, I mean that it’s missing ~10 years of commits from the already-outdated current Wikidot. Even if we are fine losing the handful of changes that Wikidot has made in the last couple of years, this is already behind that point.
When bluesoul was able to get a local instance running, we were ecstatic. The undocumented and outdated gabrys repo had come to life! You can see his extraordinary work in the very early commits of the Wikijump repository, and the subsequent work was primarily focused on modularizing and cleaning up all of the basic setup (e.g. Docker-ization, packaging PHP dependencies). We felt like we were making concrete work towards the goal of an independent platform!
Which brings me to the first misstep. In retrospect, it’s clear we should not have pursued a fork.
As a terminology clarification, a fork is when you take an existing open source software project, make your own copy of it, and starting adding changes to it that weren’t present in the original, eventually creating your own “sibling” version of it. Making a standalone replacement that lacks heritage from the original is not a fork.
While we were able to get a basic instance running, as we began work, the PHP turned out to be barren. Massive important modules were missing, unimplemented, or only partially there. For instance the Text_Wiki package, which contains Wikidot’s horribly inefficient parser, was missing very common pieces of syntax, and obviously was far from being able to render all our complex real-world pages. Many flows such as creating an account did not work.
So from the very start we were shackled with a combination of two bad situations. First, we had tons of legacy code from Wikidot’s arbitrary conceptions and constructs. We planned to remove them, but to do so iteratively rather than a wholesale rewrite. And second, we didn’t have any working platform to start with.
Normally when you work on a very old codebase (what programmers like to call “legacy“), while it can be cumbersome to move about in and make good changes, you have a system which at least (partially) works.
Here, we had neither a system that worked, nor the flexibility of a brand new project to make large-scale changes as called for. This was a bad combination which sunk many developer-hours. We solved a number of very difficult problems involving gluing this ancient PHP web server to a modern stack. But ultimately it didn’t matter because a lot of it was spinning our wheels, as the approach as a whole was in retrospect, not useful. And the lack of productivity probably contributed to developer burnout.
All this effort could’ve instead been put into directly constructing new code to achieve the same function, and it is reasonable to believe we would’ve created a minimum viable product (MVP) in that time instead. The “backwards compatible” approach, in hindsight, was a mistake, as the Wikidot PHP was effectively nonfunctional so “breaking things” didn’t really matter. There was no “live environment” which could have been affected by such changes, so backwards-compatibility was not a helpful mindset to have here.
This now informs the current strategy (see below), where all PHP code is reference only, not to be run or modified, and is discarded from the repository when it is replaced.
Another issue brought up is PHP itself. While we benefited at the time from having a developer well experienced in it, and thus able to be productive, PHP ultimately is not an amazing language to start a new project in if you can avoid it. It can be quick to iterate in to start, and enjoys a plentiful library ecosystem, but its lack of typing and some language structures can make some tasks more difficult than they need to be. This was compounded by the strong desire to glue things with the existing Wikidot codebase, which brings up the legacy issues mentioned above.
The issues with PHP are comparatively minor, to be clear. Well-written PHP can definitely make for an acceptable codebase. In fact, well-written PHP looked positively utopian compared to most of the nonsense we found in the Wikidot repository.
Implementation Issues
In addition to missteps with regards to development approach, it has also been difficult to implement the plans we have had. A reality of this situation is that we are all human. Real life, mental issues, and other distractions can sap time and energy from this project.
I’ll only speak for myself here, but 2022 especially was not a good year. I was between jobs and dealing with a number of mental health issues. In the face of some frustrating obstacles (discussed above and below), executive dysfunction has at times made focusing on meaningful contributions very difficult. Ironically, this blog post, meant to update the general public on the state of the project, has taken longer to finish up than I originally planned. This also came at a time when, for various reasons, some of the other core developers have needed to lessen their activity.
Additionally, for those unaware of the various happenings within the SCP-EN branch, the last two years have been a ridiculous whirlwind of nonstop fires and other major events. Being an EN administrator during all this, it heavily cut into the amount of time I was able to devote to development, something which a developer challenging similar problems would not have to deal with.
This chaos was not conducive to development work. In several cases I didn’t have much of a choice but to help, but at the end of it, massive incidents on EN kept on happening and will keep on happening. It’s not good for my mental health, or my progress on Wikijump if I’m constantly obligating myself to working on all of these issues. So as part of my changes, I made an O5 activity update post explaining that I am setting better boundaries, and that my staff work from here on out will need to technial, with a limited exception for temporary emergency activity.
Overall, while from a casual outside perspective it seems the project has been going for some time and is inherently fruitless, the reality is that it was equipped to handle an obviously-solvable issue but ran into an unfortunate convergence of several complications at the same time.
Revised Plan
With the prior situation having been laid out, this final section will describe the new, revised plan and how it aims to deliver a continually-developed platform.
While Wikijump began as a fork of the gabrys repository, it is no longer proceeding as one. In PR #1223 (part of WJ-1077), the whole repository was reorganized. All existing Wikidot code has been moved to legacy/, where it can be referenced (and deleted once replaced), but is not actually run or used. Likewise, PHP code is no longer an executable component of the project. The new frontend and web server will be served via SvelteKit, which appears as the new frontend service framerail/.
We will not get caught up in worrying about hypotheticals that stand in the way of more practical issues. Having shed ourselves of Wikidot legacy issues, the main goal is to iterate quickly and frequently. Design should be deliberate and care about the future, but also rapidly ship. Our build and deployment instrastructure should facilitate this, allowing changes to go quickly from commit to pull request to dev.
This has been something that I’ve thought about for some time, since it was clear our current approach was not working, and developers had begun to burn out. This new plan focuses on incremental improvements to direct features rather than a large-scale refactoring or restructuring effort, and making visible changes which can provide enthusiasm both within and without the development team.
There are a few main components within the project that are worth bringing attention to. I’ll give the name, basic description, and its current status:
| System | Description | Status |
| Backups | Non-Wikidot copy of site pages, votes, etc. | This is in-place. A backup tool has been running for a few months, which targets the most exhaustive list of SCP Wiki branches and related sites that I could find. The goal is to use a recent version of this data as part of the import process (the tool for which is in progress). |
| framerail | Web server, frontend | We need a basic setup wherein the web server takes a request, does routing, is able to reach DEEPWELL and other services, and returns the response. There is also some frontend work that needs updating. |
| DEEPWELL | API service, primary backend logic | This service is well-developed and should not take a ton of tuning to serve a basic Wikijump instance. The primary work we need here is connecting this up to the other component services to complete a full pipeline. |
| ftml | Parser and renderer library | This library has been in place for a while, but has a few notable items of development remaining before it is production-ready. |
| Infrastructure | Server hosting, deployment, etc. | The final production infrastructure plan is not ready. We are focusing on immediate progress, so we need to get a basic deployment system to dev that allows for rapid iteration by pushing to the main develop branch. Under the PHP plan, we had a real development environment which was deployed to automatically; we need to revise this for the Framerail plan. |
Additionally, we will work on providing more frequent updates, both here and in chat, on how work has been proceeding.
Final Thoughts
I know it’s been a while since 2020. Some huge events have affected the SCP community, and any EN users will have likely been impacted by the many difficult things that branch has been through in the last two years.
But I’d like to note I have not given up on the project, and strongly wish to see it through to completion. It can be hard for non-developers to tell what the status of a project is, and appearances can sometimes be deceiving. Our issues with the project, rather than being technical, have primarily been very human, with matters obvious in hindsight or issues affecting us as humans. The main challenge going forward will continue to be ensuring the project can allow all contributors to give their best effort. I do not want this important effort to die.
