An Architect's View

CFML, Clojure, Software Design, Frameworks and more...

An Architect's View

State of the CF Union survey results - my thoughts

February 11, 2010 ·

Michael Smith has blogged his thoughts about the state of the CF union survey results. As might be expected from Michael, it's a bit of a "warm fuzzy" reading of the results and a couple of commenters there cautioned against reading too much into the results given the very small number of respondents (compared to the known size of the CF user base). I've been watching the survey results with interest so I figured it was time to post my thoughts. First off, the survey sample is very small. 730 respondents. That's less than 0.1% of all CFers. Second, this represents primarily the CFers that could be reached by a conference organization and blogs / Twitter. That means that it is a distorted sample that by definition leans toward those folks most active and most likely to be either more experienced CFers or those trying to expand their skillset. That all said, let's take a look at the numbers...Many people are using more than one engine or more than one version of CF. The vast majority are using CF8 and nearly half are already using CF9. Given our sample, that's not surprising (nor that nearly a quarter of respondents are using Railo). We'd expect this group of respondents to be more leading edge than the vast majority of CFers. I would hazard a guess that a much larger percentage of the total CF population are still on earlier versions than the survey would indicate. Still, the numbers are encouraging. Framework usage is still relatively low, even in this 'leading edge' sample with Fusebox out in front at 35% (not surprising - it's been around the longest) and Model-Glue and ColdBox almost tied for second place at around 15%. I'm very pleased to see FW/1 at nearly 11% but, realistically, that just serves to underline how unrepresentative the survey results are (I'd love to be able to extrapolate that 70,000 CFers were using FW/1 but that would be somewhat at odds with the 2,000+ downloads from RIAForge! :) The vast majority of respondents (nearly 80%) have been using CFML for over six years. Almost no one has been using it for less than two years. We could conclude that almost no one is picking up CFML as a new language ("OMG! ColdFusion is dead... again!") or that, once again, the survey didn't reach a useful segment of the CF population. After all, Evans Data Corporation's (EDC) number indicate that the number of CFers has doubled over the last few years which means by definition about half of all CFers must have used CFML for less than two years! Mind you, even amongst the bleeding edge, long-time CFers in the survey, still only 25% have used OO for six years or more. That's very telling (and disappointing, in my opinion). Michael's warm fuzzy take was "over 90% use object orientation"... get real, dude! The multi-lingual aspect (another warm fuzzy from Michael) is also questionable. Yes, about a third of this small, leading edge group of CFers use Flex. That should not be a surprise. Nor that a third use Java and/or PHP (with about 20% using .NET). A small percentage claimed experience with other 'hot' web languages. I wouldn't really call the majority of CFers multi-lingual, let alone "heavily multilingual" (per Michael). No surprise on the DB question: SQL Server and MySQL were far and away the most popular (two thirds using MySQL was higher than I expected tho'). Whilst SVN is popular (just over 60%), it was indeed a shock that nearly a quarter of users do not use a source code control tool (sorry, zipping up folders is not version control!). And this is amongst the top 0.1% of all CFers... On User Groups, Michael again went for a warm fuzzy ("Two thirds of developer attend a local user group some of the time") when the reality is that over a third never attend and over half attend at most once a year (no doubt when Ben or Adam roll through town). Less than a quarter of this leading edge sample of developers attend more than half a dozen times a year! Boy, are you folks missing out on great opportunities for networking and learning with almost no cost to yourselves! Finally, on top challenges, I guess it's no surprise that we're all a bit overwhelmed with "too much work, too little time" as the #1 issue (nearly two thirds). Nearly half of us don't like other people's code(!) and almost a third find learning each new version of ColdFusion to be their biggest challenge. Nearly a quarter say their biggest challenges are crashing servers and/or security problems. That should definitely concern us. So, overall, a fun and interesting set of results that shouldn't be taken too seriously. I do think a couple of the answers should ring alarm bells given the sample likely represents the leading edge of our community. I'm very disappointed that the survey didn't reach a larger audience and cover a lot more new-to-CFML developers but that seems to be a hard group to reach. What do you think about the survey?

Tags: cfunited10 · coldfusion

17 responses

  • 1 John Wilker // Feb 11, 2010 at 9:55 AM

    I wonder, did Adobe help promote the survey? Obviously it's biased towards CFunited, in that I'm sure that data is mined, but I agree, the data set is too small to be truly useful.

    Even if Adobe can't/won't support Michael's survey, it'd be nice to see them do one. Heck do one for both CF and Flash/Flex.

    Throwing out big ol' numbers is nice, but it'd be nice to know more about what makes those numbers up.
  • 2 John Farrar // Feb 11, 2010 at 9:58 AM

    I think you hit the mark. One of the best things a survey like this does is to reveal if there are problem areas in the core users. (Core being up for grabs in definition of course.) The most scary part of the survey was how many people are in this group using Access as a database. Shivers should crawl through our being on that one.
  • 3 Ryan Guill // Feb 11, 2010 at 10:45 AM

    I didn't take the survey myself mostly because there was no way to opt out of the marketing information from CFUnited. I agree though, I would really like to see this information from a much larger sample size. Too bad Adobe can't do a big one, maybe with some incentive to fill it out.
  • 4 Joe Rinehart // Feb 11, 2010 at 11:03 AM

    > What do you think about the survey?

    Bluntly...I think it's misrepresentative. You pointed it out well, but I'd go a bit further to say that labeling such a small sample of the CF community the "State of the CF Union" is misleading, and probably a bad reference for those planning CFUnited 2010.

    I really did enjoy your analysis, and I think that your insights were the most valuable bits to come out of the survey's result.

    Wow, that came off as a bit of ass-kissery. Oh well.
  • 5 Nathan D // Feb 11, 2010 at 11:29 AM

    Statistically that's a good sample size, though the selection bias sounds very real. In my experience a huge percent of CFers toil in obscurity building workgroup apps inside big companies.
  • 6 Jim Priest // Feb 11, 2010 at 11:41 AM

    Nice summary!

    We the community (and Adobe) really need to figure out how to reach that other 99.9%...

  • 7 Kerr // Feb 11, 2010 at 12:23 PM

    The best one can do with these results is speculate as to what this means from the top down view, which you've done well. I find the general lack of use of revision control quite alarming.
  • 8 Allen // Feb 11, 2010 at 3:00 PM

    Not that I was there to be part of this survey but as food for thought, I usually just give random answers to these sort of things on surveys.
  • 9 James Morrow // Feb 11, 2010 at 3:04 PM

    It's definitely shocking/dismaying to see so many people not using source control... and how can your possibly justify still using Access as your DB?

    I'm kind of at a loss to explain either of these answers - sheer ennui? Actively resisting change of any kind?

    How can the benefits of using version control not be self-evident - it takes almost no time or effort to setup and configure SVN or CVS, and doesn't take much hardware either.
  • 10 Andy // Feb 11, 2010 at 3:21 PM

    @John - sometimes we use Access because we have no choice or is the best option. For instance we have customers who have large custom applications that are running (and working perfectly) on DOS based versions of FoxPRO, dBASE, etc.

    In order to add functionality and/or access data without touching the present app and/or interrupting the current work-flow, we can use Access to attach multiple dbf files (dozens to hundreds) so that we can use a single DSN in CF to add functionality to these applications without ever mucking around with their working systems.

    I'd love to hear about other ways people handle these types of situations when the customer cannot or does not allow the main app to be touched?
  • 11 Adam Presley // Feb 11, 2010 at 9:12 PM

    Although I find numbers like these can be easily misleading, and manipulated to any angle, I do find the stats about user group attendance to be interesting. As a regular attendee of the Dallas CFUG I find the attendance varies from "lots of faces" to "very few faces". I have to agree that many are missing out on great opportunities to network and great learning.
  • 12 Martijn van der Woud // Feb 12, 2010 at 12:44 AM

    @Nathan Hey that's nice, a CF developer who can actually talk sensibly about statistics! ;)

    100% agree with Nathan's remarks on sample size and selection bias.
  • 13 Dan Fredericks // Feb 12, 2010 at 3:16 AM

    I think one other thing to look at is where do these people work, government, or private. Government sometimes can keep versions and code for years...the it ain't broke don't fix it approach. Or you might have old school managers that don't know how important updating is.
    Overall the survey gives an idea of what people are doing, and how many people wanted to take the time. Maybe all conferences should have a survey booth where people can come by and fill this survey out. Make it a year long survey...maybe then a few thousand will fill it out....oh, and use online, we are tech people and should be able to find a good way to get this out to the masses...overall it was interesting.
  • 14 Richard // Feb 12, 2010 at 8:34 AM

    @Nathan. You're right - at least you are from where I'm standing. There may be a reason why people working for big corporations don't contribute to surveys like this but it's a shame for CF. Having said which I have to confess I'm a guilty party myself.
  • 15 Kerr // Feb 13, 2010 at 6:06 AM

    I'm not sure what that reason would be, other than a lack of wanting to keep up with the technologies used at the workplace. I work for a very large corporation, and have come across many developers in my travels. I'd say maybe 10% I've worked with have the "seeker" bug, if you will. The vast majority succumb to complacency and fear of change, which comes to bear in resistance to using new (to them) things like version control, frameworks, coding standards, etc. I'm not saying it makes sense, but that's just been my experience.
  • 16 Allen // Feb 13, 2010 at 8:58 AM

    While there are times when not putting money into upgrading an application isn't wise, we shouldn't assume it's the wrong thing to do. If you have a half dozen users using some web app they built in 1999 that hits access, what are the chances that spending $50-100k (keep in mind labor, testing, etc) to upgrade it to use MS SQL Server and CF9, are they really going to ever get that money back? If anything, it touches on the importance of writing code that doesn't need to be touched once it's deployed.
  • 17 Jon // Feb 16, 2010 at 4:08 PM

    Reading some of the comments it looks like that if the results are not what one expects, the survey is biased, but if they would have been what was expected then the survey would have been OK. mmmmmm.