How we keep the map ad free.

Hello. We thought it might be interesting to talk a little bit about what goes into the general development and maintenance of the pinball map. Specifically, the things that we do to keep everything free (and ad free) for you, our treasured users.\

The Perl Years

Initially, the PBM was just the “Portland Pinball Map”, because that’s where Ryan and I played pinball. It’s just the town we knew, OK? There was a local Google Map, but it was difficult to work with. So, Ryan and I decided to replace it with something more user friendly. This was in 2008. The Portland Pinball Map was written entirely in Perl/HTML/CSS/JavaScript. My background at the time was pretty heavily Perl-based. So, that was just the language that made the most sense to me.

Getting a website like the PBM up and running basically boils down to these things:

Web Framework:} Like I mentioned before, the initial map was written mostly in Perl. A lot of people don’t like Perl, and find it horrifying that someone would use it for anything except scripting. And hey, I get that, but I’ve seen people do great stuff with Perl too, so lay off. In addition to Perl modules for our Models and Controllers, we also had Mason embedded in our HTML.

Database: Initially, the PBM ran on top of MySQL. This was before Postgres really captured my heart, and I really didn’t care what database we used as long as it was relational.

Hosting: Here is where an actual cost comes in. In order to host the map at this point, we used Dreamhost. Dreamhost offers a plan for $120 a year that gives you unlimited bandwidth, disk space, and one free domain. It wasn’t a bad deal at all. You get a virtual server that isn’t very beefy, but we also weren’t really dealing with a heavy traffic load at the time. And it’s just $10 out of my pocket every month, which is not a big deal at all.

Domain Registration: Dreamhost gave us one free domain, so the costs here were rolled into the $120 per year fee.

Data Administration:` This was handled 100% by Ryan. Since we were only dealing with Portland at this point, Ryan was able to use his extensive knowledge of the city to make sure that the data on the site was accurate and clean.

We were able to keep our out of pocket costs down to $120 a year by going entirely with Dreamhost. We were able to eliminate any costs of the software itself by staying 100% with free / open source tools. And, we were able to eliminate any “human costs” by just donating our time to it in the form of software development and data administration.

BUT, the most important component was, of course, the users. The site would be completely empty if you guys weren’t out there playing pinball and then telling your fellow ballers where the machines are.\

Mobile Times

In early 2010, we got approach by another developer with the idea of expanding the map to mobile devices. This was an area that no other pinball maps were exploring, so we thought it was worth a shot. Isaac Ruiz (of brackelope fame) put together the app for us.

This is maybe a story for a different blog post, but this is where a lot of my current technical beefs with the website began. A lack of inexperience on my part in API design, coupled with the technology stack we were using made for some hasty decisions that continue to haunt us to this day.

When the iOS app first launched, Isaac chose to sell it for $1. The economics of mobile apps are sort of goofy. People pay $500 for a phone, but freak out when an app is a dollar. But hey, I get it. Isaac made the decision to sell it based on the fact that Apple charges developers $100 a year just for the honor of being an Apple developer. A year or so ago, Isaac made the app a free download (he is stacking bills with brackelope now, so…). At the same time, he donated $200 of the app proceeds to the pinball map so that we could register pinballmap.com (more on this later), paid a designer Drew Marshall for work that he did for the app, and also sponsored some local pinball tournaments with the money. So, even your one dollars were ultimately going back into the pinball community in some form.

To follow-up on the success of the iOS app, I took this opportunity to develop an Android app for the map. This was sort of a cool project for me because it gave me the chance to “get my feet wet” with mobile development for the first time. It’s much easier approaching a project like this when you have an audience, a clear set of requirements, and the desire to learn a new thing. On a personal level, this is where the synergy of the pinballmap and my development interests come together. I enjoy exploring new programming languages/frameworks, and it’s a lot easier to get motivated to do this exploration when I know that the results of my efforts aren’t going to get shelved in a repository when they come to fruition. So, thank you for your patience with issues that come up on the map, a lot of stuff here is a learning exercise for me.

Anyway, roundabout when we had both apps launched, we noticed that the traffic to the site more than doubled. That was exciting, but scary. Even though hosting and bandwidth was not an issue, performance was beginning to be.

Added costs:
Another developer added to our team, doing unpaid work.

A whole new breed of user, now doing the important community service of taking their cellphones out in the middle of a bar and selecting a machine from a drop down menu.

The Expansion Teams

In 2010, we heard the call from our brethren in the north. They wanted a pinball map too. We were a little nervous about expanding, mostly because we didn’t want the data to go untended. But, we put our trust in the enthusiasm of the administrators and users in other areas, and we think it’s paid off! We started by adding LA/Seattle/BC, and saw our traffic go up by a factor of more than 5.

After the initial swath of regions rolled in, we continued to slowly expand the map. Once we added “bay area”, we knew that we were about to be in trouble. Traffic was way way up, and performance was starting to take a noticeable hit.

A pretty classic way to address problems like this in computer science is to “throw RAM at it”. And, that (maybe) would have worked, and been awesome. But, that also would have cost money, and we knew this wasn’t an option unless we started putting ads on the site.

Added costs:
Many many more admins; all volunteering their time to make sure that data in their areas was well-formed, and maintained.

Supporting thousands more users; all volunteering their time to make sure that their communities knew where to put their quarters.

Ruby Times

With the increase in site traffic, we looked to a modern web framework to split up the load more intelligently (and give us access to tools that Perl didn’t have). This happened to coincide nicely with a new project that I was working on at my 9-5 job; a rails app. I thought it would be mutually beneficial to work and the PBM audience if I was learning rails in two places at once. I’m not sure if that ended up being the case, but here we are. Porting the app from Perl to Ruby took a long time, but was actually sort of fun. This smart guy I know at work once told me “the best way to learn a new language is to port something to it that you are already familiar with”, thanks Ian. I took it to heart, and tried to do the best I could.

This iteration of the PBM was open-sourced on github, to try and encourage community participation in the project. So, if you are a rails developer or would like to become a rails developer, maybe check out http://www.github.com/scottwainstock/pbm and see what you think.

Although the spirit of the PBM remained the same after the port to Ruby, the technology underneath it had completely changed. In order to keep costs the same, we continued to host the site with Dreamhost. Unfortunately, at the time, Dreamhost really wasn’t setup to host rails apps. So this meant that we add a couple goofy hacks to keep things running (e.g. a few nasty symlinks, lack of passenger, and a slightly convoluted Capfile). It was around this time that we also noticed that Dreamhost had a lot of issues with uptime. We were seeing the site go down a couple times a week, sometimes for as long as 2 hours at a time. We knew that pinballers needed constant connectivity with their data, and we knew that we needed to make a major change if we were going to be able to offer this.

Added costs:
Isaac donated $200 to secure the pinballmap.com domain. There is an on-going $30 a year fee for this.

The Heroku Years (present times)

We saw great gains with Ruby. Performance was way up, code was easier to maintain, and deployments were a snap. Starting over and writing (almost) everything from scratch let us get rid of a lot of rough edges that were starting to be an issue. Things were almost totally awesome. We were falling way short in one area though, Dreamhost. Dreamhost continued to have outages issues. At one point the site went down seven times in one week, and was unavailable for up to an hour at a time for each outage. We were started to get complaints, and we knew there had to be a better way. Enter Heroku…

Heroku is essential cloud hosting (and more) as a service. It’s tailor-made for the rails app that the pinball map had evolved to. It also presented us with a number of hurdles that we had to overcome in order to keep the monthly fees of running the site to a minimum. Specifically…

A 10,000 row limit on the size of our database. In order to use Heroku’s free-tier of postgres, we have to make sure that the database remains under 10,000 rows. In order to do this, we eliminate a lot of history from the system. So, when you remove a machine, we actually are removing the data from the database altogether. Ideally, we’d retain history like this. (don’t stress though, we have daily backups of the entire system, so the data is available, if not immediately available) Even with a fairly aggressive “trimming of the fat”, the system currently hovers around 9000 rows.

Goofy logging. Heroku doesn’t really allow us to write files to their server (outside of when we deploy the code). Because of this, we use a nerfed version of Papertrail for our logging. What this means is that we only get to see the first 10 megs of each day’s logs. Despite a number of optimizations in the nature of the data we actually tore, this 10 megs only gets us through to about 4pm each day.

A limp version of New Relic. New Relic is an application performance monitor. It’s a really cool service that shows you information about the health of the system over time. Unfortunately, the free version doesn’t show us much at all. So, we’ve had to switch over to Google Analytics to get an idea of what sort of load the system is under, and GA doesn’t really do much in the way of actual performance metrics.

Outsourced email. We use Sendgrid, which is a pretty great email service, don’t get me wrong. The problem with their free service is that we’re limited to 200 emails per day. After that, silence. We only process maybe 50 legitimate emails a day (in the form of new location suggestions, machine additions/deletions, etc), so the 200 limit typically isn’t an issue. We’ve had three or four outbreaks of spam throughout the years though, and have had to write some code to check for robots.

S3 for image hosting. Like I mentioned before, we aren’t allowed to upload files to heroku beyond the code that runs the actual site. So, when we moved things there we needed to open up an S3 account to host the images of the locations that you see on the map. This only costs us a dollar or so a year, but it’s sort of a pain to not just dump everything locally.

We have a lot of processing restrictions with the free instance of Heroku. The site runs on 512 megs of RAM, has a Max 200msec per second CPU time, a limit of 100 megs for size of the app, and a limit of 2tb per month in traffic. I don’t think we get anywhere close to 2tb a month in traffic, but we’re constantly bumping up against the RAM and processing restrictions (in addition to the database limit). What this has meant for us the need to constantly “run lean”; looking for code optimizations and caching solutions where ever we can. On the one hand, it’s been frustrating, on the other hand it’s sometimes fun to see how much we can squeeze out of how little we have.

I don’t want you to take this the wrong way or anything, we love Heroku. It’s hard to complain about a service that is completely free, and rock solid. The site offers a ton of useful stuff that “just work”, and we’ve never had to pay them a penny for any of it. I just thought it’d be interesting to call out the sort of goofy workarounds that we need to do in order to keep things ad free. Even just $50 a month would make a huge difference in the speed and responsiveness of the site. But, as things stand right now, the site runs for about ~$2 a month, which all goes to the domain registration fees. Heroku enabled us ditch the $120 a year Dreamhost fees. Not bad!

It’s worth calling out one more time that the number one thing that keeps things running around here is the input of the users. We are here to provide a site that is responsive, intuitive to use, and robust in features. But without you providing the data, none of this code does anything useful at all. So, thank you for that.

I think that’s it for now. If any of the technical bits in there caught anyone’s eye, let me know, and I’d be happy to expand on them. I tried to keep this post constrained to cost-centric issues, but I love typing about this stuff in general. Thanks for reading and thanks for the continued support of your pinball maps.