June 3, 2014

Getting Started

Sometimes it's very enlightening to look back at the beginning of a project to see how things got set up and how we started down the path that led to where we are today. In this post, I'm going to talk about the first ten tickets we created at World Singles as we kicked off our green field rewrite project five years ago.

I've been involved with World Singles for about five years now, about three and a half years as a full-time engineer. The project was a green field rewrite of a dating system the company had evolved over about a decade that, back in 2009, was running on ColdFusion 8 on Windows, and using SQL Server. The new platform soft-launched in late 2011 as we migrated a few small sites across and our full launch - migrating millions of members in the process - was May 2012. At that point we switched from "build" mode to "operations" mode, and today we maintain a large codebase that is a combination of CFML and Clojure, running on Railo 4.2 on Linux, and using MySQL and MongoDB, running partly in our East Coast data center and partly on Amazon.

Like all projects, it's had some ups and downs, but overall it's been great: I love my team, we love working with Clojure, and we have a steady stream of interesting problems to solve, working with a large user base, on a multi-tenant, multi-lingual platform that generates millions of records of data every day. It's a lot of fun. And we all get to work from home.

Sometimes it's very enlightening to look back at the beginning of a project to see how things got set up and how we started down the path that led to where we are today. In this post, I'm going to talk about the first ten tickets we created as we kicked the project off. Eleven if you include ticket "zero".

  • #0 - Choose a bug tracking / ticketing system. We chose Unfuddle. It's clean and simple. It's easy to use. It provides Git (and SVN) hosting. It provides notebooks (wikis), ticketing, time management, customizable "agile" task boards, collaboration with external users, and it's pleasing to the eye. I've never regreted our choice of Unfuddle (even when they did a massive overhaul of the UI and it took us a week or so to get used to the radically new ticket editing workflow!).

  • #1 - Version control. Yes, really, this was our first ticket in Unfuddle. The resolution to this ticket says:

Selected vcs system (git), created repository in Unfuddle, and provided detailed documentation on why git, how to set it up, how to connect to the repo and how to work with git.

  • And the documentation was all there in an Unfuddle notebook for the whole team. A good first step.

  • #2 - Developer image. Once we had version control setup and documented, we needed an easy way for every developer to have a full, self-contained local development environment. We had some developers on Windows, some on OS X, some on Linux, so we created a VMWare image with all the basic development tools, a database, a standardized ColdFusion installation, with Apache properly configured etc. This established a basic working practice for everyone on the team: develop and test everything locally, commit to Git, push to Unfuddle. We could then pull the latest code down to a showcase / QA server for the business team to review, whenever we or they wanted.

  • #3 - Project management system. Although we had bug tracking and wikis, we wanted to nail down how communication would work in practice. We created a project management mailing list for discussion threads. We created a notebook section in Unfuddle for documenting decisions and requirements. We decided to use Basecamp for more free-form evolution of business ideas. We agreed to use tickets in Unfuddle for all actionable work, and we settled on a Scrum-like process for day-to-day development, with short, regular sprints so we could get fast feedback from the business team, and they could easily see what progress we were making.

  • #4 - General project management. Since we had agreed to use Unfuddle for time tracking, we created a ticket against which to track project management hours that didn't fit into any actual work tickets. We used this for the first six months of the project (and logged about 300 hours against it).

  • #5 - Performance planning/tuning. This was mostly a placeholder (and initially focused on how to make a Reactor-based application perform better!). It was superceded by several more specific tickets, six months into the project. But it's one of those things we wanted on the radar early for tracking purposes.

  • #6 - Architectural planning. Like ticket #4, this was a time tracking bucket that we used for the first six months of the project.

  • #7 - Set up Continuous Integration. Yup, even before we got to our first actual coding ticket, as part of the early project setup, we wanted a Continuous Integration server. Whilst we were using ColdFusion for local development (prerelease builds of ACF9, at the time), we chose to use Railo 3.2 for the CI server so that we could ensure our code was cross-platform - we were still evaluating which engine to ultimately go to production with. The resolution of this ticket says:

Apache / Tomcat / Railo / MySQL / Transparensee / Hudson in place. Automated test run restarts Railo, reloads the DB, reloads Transparensee, cleans the Reactor project, runs all test suites and generates test results.

  • We developed an Ant script that stopped and started Railo, tore down and rebuilt the test database, using a canned dataset we created (with 1,000 random users), repopulated the search engine we use and cleaned up generated files, then ran our fledgling MXUnit test suite (and later our fledgling Selenium test suite).

  • #8 - Display About us/trust. This was our first actual code ticket. The company had selected ColdBox, ColdSpring, and Reactor as our basic frameworks (yeah, no ticket for that, it was a choice that essentially predated the project "getting started"). This ticket was to produce a first working skeleton of the application that could actually display dynamically generated pages of content from the database. We created the skeleton of the site navigation and handlers for each section as part of this ticket. The "trust" in the ticket title was about showing that we really could produce basic multilingual content dynamically and show an application architecture that worked for the business.

  • #9 - Implement resource bundles for templates. And this was also an early key requirement: so that we could support Internationalization from day one and perform Localization of each site's content easily.

  • #10 - Display appropriate template for each site. This was our other key requirement: the ability to easily skin each site differently. Like #9, this was an important proof of concept to show we could support multiple sites, in multiple languages, on a single codebase, with easy customization of page layouts, content, and even forms / questions we asked.

So that's how we got started. Bug tracking, version control, local development environment, continuous integration and the key concepts tackled first!

A reasonable question is to ask what has changed in our approach over the five years since. We're still using Unfuddle (in case you're wondering, we're up to ticket 6537 as I write this!), we're still using Git (and still loving it). Our development stack has changed, as has some of our technology.

Over time we all migrated to Macs for development so maintaining the VM image stopped being important: everyone could have the entire development stack locally. We eventually settled on Railo instead of ColdFusion (we're on Railo 4.2 now), and we added MongoDB to MySQL a couple of years ago. We added some Scala code in 2010 to tackle a problematic long-running process (that did a lot of XML transformation and publishing). We added Clojure code in 2011 for a few key processes and then replaced Scala with Clojure and today Clojure is our primary language for all new development, often running inside Railo. We stopped using Reactor (we wrote a data mapper in Clojure that is very close to the "metal" of JDBC). Recently we stopped using MXUnit and replaced it with TestBox. We're slowing changing over from Selenium RC tests to WebDriver (powered by Clojure). We have about 20,000 lines of Clojure now and our CFML code base is holding steady at around 39,000 lines of Model and Controller CFCs and 45,000 lines of View cfm files.

Tags: clojure process