Always Move Fast

The Balance of Power: Business and Technology

Thoughts on Web Development: Automation, Code Generation, and Scheme

Every so often I become frustrated with my tools. I mostly spend my time building web applications and it's painfully obvious how much boilerplate code I write in Python to put live apps on the web. Python isn't alone in this respect. Any language will have some amount of boilerplate. Lately I've been spending some time on automating deployment of web applications. Any web developer worth his or her salt will have some process in place for automated deployments. Another thing worth automating is your initial project generation. The more popular web frameworks have something like this in place. Pylons, Django, Rails, etc are all capable of generating a starting point for your web app.

My own web framework, blueberry, initially had an automated starting point for web apps. It also had some code for managing database migrations. Over time, these features became somewhat constraining so I ripped them out completely. The last few web applications I've worked on used Tornado, which has zero code generation for a starting point. Not having a web app completely generated for me is a big bonus in my opinion. I'd rather build tools for automating web app generation to fit my style, rather than fitting my style to the web framework.

Another topic worth exploring is web form generation. There are tradeoffs to be made in this area and many ways of doing the same thing: building web forms, validating them, and storing the result in a database. Python has some really cool form generation libraries (ToscaWidgets, for example). There's also formencode for validating and inserting defaults/errors into your forms. Using ToscaWidgets and formencode together is pretty cool if you can get it working perfectly. ToscaWidgets is especially complex. At the time I was using it the docs were a little light, so I spent most of my time inside of the code learning how it worked.

Then there's the other side of the coin: Using straight HTML to build your forms. You can 100% use formencode for validating your hand-rolled forms, and it works really well. By the time you get into displaying errors, inserting defaults, etc, you'll start wishing you went with a form generator. Hand-rolled HTML for anything complex is A Bad Idea. I mean, unless you like feeling unproductive when you write your thousandth signup form.

There's also this new-ish trend of using jQuery to generate all of your markup. I'm not sure how I feel about this. Using jQuery is awesome and it's a great piece of technology, but it's possible to make your jQuery markup generation code really complex and it can even take up more of your editor space than your normal markup. That tradeoff may or may not be worth it to you.

Focusing back on server-side technologies, I think there is a lot to be learned from Arc and Scheme in general. I keep coming back to using scheme for my web projects, but a big problem I have is that there's no clear way to do it. With LISP there's Hunchentoot, but even then you have to install all kinds of libraries and it may not even install properly. I used to use PLT Scheme (now Racket) and even wrote a few articles on using it for web development. PLT is in a world of its own. It's an advanced Scheme that has tons of libraries. It even comes with its own web server.

Using languages like Scheme, if done correctly, your web app becomes a set of high-level, short functions that were built from the ground up to limit the amount of boilerplate you have to write. When I'm writing Python I try to incorporate this way of thinking into my own software. To do that really well, you almost have to create an entirely new programming language on top of the language you're using. That's what Arc does. Python doesn't make it easy to do that, but it can be done.

Lately I've been spending some time looking at other Schemes and toying around with Chicken Scheme. I'm hoping to get some web apps running with it one of these days. Just last night I wrote some C code and used SWIG to interface Chicken and libevent. It wasn't a complete interface to libevent though. I made a super simple timing event and called it from Chicken. I'll probably end up writing some simple servers in libevent and using them in Chicken. Maybe even a new web server or tying Chicken Scheme into Mongrel2.

Form Store API Available for Testing

Update: I modified the limits to monthly instead of daily -- 1 database, 500 entries per month, 1,000 primary key queries per month.

I spent most of today working on getting the Form Store API ready for beta testing. I added a really simple page of documentation on how to do basic things with Form Store. At the moment all you need to get an API key is an email address. It will create a free account for you automatically. You can't login or anything yet, but I figure when that time comes I'll send out emails asking you to enter a password to gain access to the admin tools.

When your API key is generated, a 14-day secure cookie is set in your browser so make sure you save your API key somewhere safe in case your cookies are cleared. After the key is generated, javascript will update the `curl` commands on the page to reflect your new API key.

There are API limits, but they aren't set in stone. Expect me to update the limits as I get a better feel for how Form Store is being used. I may add data transfer limits and disk space limits, but for now those are nonexistent. For testing purposes, the free accounts are limited to 1 database, 500 entries per month, and 1,000 primary key queries per month.

If you send me an email at david@alwaysmovefast.com and ask me to increase your limits for testing, I'll happily do it. Form Store is really basic right now, but I have a lot of things planned for it. I'd love it if people would hammer on it a bit and break things.

What can you do with Form Store API right now?
  • Create a new database
  • Insert new records
  • Get records by primary key
  • Delete individual records
  • Delete databases
In the near future I'll be adding API support for
  • Modifying existing records
  • Range queries by datetime (all records are indexed by a `created` timestamp automatically)
  • Multi-key queries
  • Database exporting
I'm also working on a dashboard/admin interface to help users manage their data, do some reporting, exporting, etc.

Check out the Python Form Store API examples as a supplement to the `curl` commandline interface.

Tools for Developers

While considering many different ideas for projects I wanted to work on, I felt that I needed to solve some of my own problems (as a developer) first. Looking back on my own career path and development strong points, I realized that the projects I've had success with have been tools that made my life easier. Take SimpleStore for example. I wanted a simple way to store schema-less data in MySQL and that's what I created. Many of my projects, especially ones I haven't open sourced yet, are geared towards streamlining my own development processes.

I could spend tons of my time working on projects like mobile apps, games, etc, and never come up with something significant or useful. My core competencies seem to be in tooling and automation. I'd like to take some time and describe what I think may be the future of Always Move Fast and this blog.

I've been working on another schema-less datastore that is accessed via a REST API. The original idea was to create a simple database that could be hooked up to web forms. It would allow webmasters and developers to focus on their UI without having to worry about setting up a database, defining a schema, or modifying that schema when they change their forms. I've been having a couple friends test the API out a little. While I don't have a set-in-stone web interface for it yet, I'm going to be opening up the API to developers who want to hammer on it for free.

Something that schema-less databases are a good fit for are spreadsheet representations. I'd like to have a simple version of this concept developed soon so it can be part of the data storage offering.

One of the other tools I've been working on for myself is automated deployment. I wrote some simple scripts to manage deploying code to my production servers and the next step I'd like to take is continuous integration. After I get some of the bugs hammered out, I'm hoping to work on some continuous integration server tech so I can have a nice web interface for managing my tests and deployments.

That's pretty much what's on my radar at the moment. I really just want to offer people the tools I use that make my life as a developer easier. In the next couple days I should have the datastore API ready to be used by anyone. For the time being there won't be any web interface to it, but that's coming soon.