hquery: an unobtrusive server script implementation

I’d previously blogged about some ideas to resolve the developer-designer gulf regarding templates. I’m happy I’ve finally found time to code up a proof-of-concept:

hquery - the unobtrusive server script from choonkeat on Vimeo.
Q: What is “unobtrusive server script”?

If “unobtrusive javascript” adds interaction to a web page without being inside the markup - then “unobtrusive server script” replaces data in a web page without being inside the markup (and design).

You’ll get a clean separation of 1) designed web page is constructed, then 2) server data is weaved in, and finally 3) javascript hooks up the interactive portions.

The key thing is that all 3 layers are stringed together by CSS path - a concept / language that both designers and developers can communicate with.

Q: Is this a silver bullet?

No. There are constraints: everything that needs to be dynamic, need its own CSS/XPath. e.g. Google’s result page markup isn’t very suitable:

  <div>this is abstract<cite>citation</cite></div>

Notice how we can’t address the abstract content without stepping on <cite>. When using USS, we’ll need the markup to be:

  <div><span>this is abstract</span><cite>citation</cite></div>

Then both abstract and citation can be targeted independently. Furthermore, to be totally isolated from markup changes, it is best to only use classes and ids without the tag names.

Q: Is USS a Ruby on Rails thing?

Nope. “hquery” is just a library I wrote to do USS in Rails. I believe any language / framework can have its own USS renderer.

Q: How do I write javascript that needs dynamic values?

Instead of writing mixed-up code like this:

  alert(“I am <%= user.name %>”);

Try doing it this way (separating data from logic):

<div id=”json”><script>var user = {name: ‘foo’};</script></div>
  alert(“I am ” + user.name);

Then in your script, you can swap out the JSON data:

select “#json script” do |ele|
  ele.html “var person = ” + current_user.to_json   

Q: Where next?

I dunno. Since hquery script is declarative, we can compile to regular <% messy %> template as runtime? Or, have build tasks that can check that corresponding HTML contain necessary elements - i.e. a new design don’t break/miss out anything?

Anything else? Love to hear your thoughts!

PS: I find myself trusting Yahoo search results when its in Google skin!

(Chew) Choon Keat

N673588474_2688loves programming and has been doing that professionally for over 10 years - moving from C, Perl, Java to Ruby & Javascript. Choon Keat is curious about everything and is always thinking of ways to improve [his] life with better design, better software. Sometimes, he even blogs about these things.

Choon Keat is practical, delivers simple solution and executes iteratively. He has been practicing that on a startup he has founded, SharedCopy.

Professional summary

Choon Keat finds himself involved in startups most of the time. But he did once venture into a mid-size J2EE shop to find out what enterprise software was about. 

His hands-on experience range from (server-side) administrating Linux boxes, writing web apps & integration with SMSC, MMSC & Jabber, to (client-side) writing Eclipse plugins, Javascript, Flash ActionScript, to programming on various other devices like phones and the TV.

His recent work revolves around email, Ruby, Rails and a lot of Javascript.

Software passion

Choon Keat loves open source. He gives back by submitting patches to the Rails project and releasing his own software as open source: hquery, oauth4r, poormans-trends, krjsweb_sg_form_builderhpricot_forms, rssfwd.

Occasionally, the need to share overflows and he turns them into talks for geek audience: Sharing session for PHP User Group, OAuth for Singapore Ruby Brigade, and Rails boot camp 1 and Rails boot camp 2.

Choon Keat understands that experiments are a large part of learning. Hence he’ll occasionally deploy experiments like Real [Twitter] Friends, Rss Paper for iPhone, Timezone UI & Translation UI just to see how well they work.

What he’s working on now

Trying to convince others to employ a image server to handle file uploads, instead of simply reaching for the most fashionable gem.


e-mail:  contactme@choonkeat.com

twitter: @choonkeat

So love this film

USA, Google IO and RailsConf '08

This is a brain dump, knowing that if I don't write this now, I probably won't ever.

They say the weather in San Francisco is perfect, but I think I'd prefer 8 degrees (celsius) warmer.. then it'll be really perfect. (And for my friends complaining their MacBook Pro runs hot… its a geographical/design problem, you see the CPU fans will never kick in here ;-)

A "quarter" is a really unproportionately valuable coin. Its a popular choice everywhere and you'll never have enough of it. I'm not sure if I can keep up, collecting 12 of them for my weekly laundry.

On the street, there is no "green man" for pedestrians. Its a myth. The man is white. Lines across a junction means nothing, but a double-yellow in the middle of the road means it is 2-way. Turn right on red light is the default.

Google is much more valuable in the States. The level of integration (and ease of use!!) they provide with real-world services is certainly something drool-worthy for Singaporeans. For example, look at this map. Something a lot of ppl I talked to didn't realise is the information on the left pane: Public Transit. It has price, train schedules (go to options & choose when you wanna arrive). A tourist like me can really survive on Google. Just add free wifi.

The night before driving to Big Sur, we went Google Earth surfing to recce the place. After been there for real, I think Google Earth redefines WYSIWYG.

Google IO
2 things impressed me: Android demo and the food.

For the 2 of you who haven't read, the Android demo in the first keynote was really impressive. Clearly iPhone inspired, but not as sheek.

A "home" screen where you use swipe gesture to scroll. Favorite apps anchored at bottom. Press and hold to drag icons around. A Webkit browser means all their previous iPhone mods for google search, mail, etc now works the same in Android.. such coincidence. I bet screen dimensions will match as well.

Best of all, a Google Maps "Street view" was demoed. It was really, really smooth. Holding it up and pointing your device north or south will actually swivel the street view to match your direction - how cool is that! Come to think of it, a compass is for street view might be a bit gimmicky. But if the compass works for google maps itself (where the maps rotate appropriately) now THAT will be cool.
[Update: This is do-able indeed on Webkit!]

RailsConf 08
My favourite part of the conference is the first lightning talk where I signed up and demo SharedCopy. This is the response in the IRC channel (roughly the same from the floor)

Response to SharedCopy Demo [RailsConf 2008]

[screenshot donated by Chu Yeow - thanks!]

It is hard to do SharedCopy justice by describing it in words. It needs a demo. I'm glad I had the chance to do a quick one in RailsConf. My only regret is that I forgot to show the read-later feature.

Now Maglev is really interesting. Avi's showmanship definitely helped (he was putting and pulling rabbits out of the hat across different Ruby VMs). But Maglev is interesting in a different way JRuby / Rubinius is. The latter 2 effectively replaces MRI. Maglev however brings the Smalltalk way of doing things with it - if you have object persistence, why use a database? i.e. A regular joe Rails app may choose to run on MRI, JRuby or Rubinius. I don't see Maglev having much incentive (and interest) to run a regular Joe Rails app other than marketing purpose. Their vision is to run Ruby/Rails apps architected in an entirely different way. Hence, their paths don't really meet.

There are gripes about the conference though. Some keynotes were really uninspiring (compared to RailsConf '06) the upside is I got my mod_rails working during one of the keynotes! And as if finding (cheap) hotels for a conference wasn't hard enough, it had to coincide with Portland's Rose Festival. Doh!

Something's Gotta Give

Saw this clarification on TC regarding Twitter
the issue certainly isn’t with creating new messages, the issue is when you check your own messages For eg, if I am following 200 people, the query needs to check the new messages from all those people and sort them into date order. That query would be really intense link »
- from Twitter At Scale: Will It Work? via sharedcopy.com

I’d heard that earlier in the Gilmor Gang : Blaine clarified that static resources (e.g. profile images) are no issue at all, and that the the bigger issue is with each person’s “home page”, where multiple streams are joined and ordered chronologically. When 1 of the connected people update, all related cached “home pages” are obsolete - hence they are generated painfully often. (so don’t refresh your own page so much!)

I’m not a scaling expert. Nor do I know anything about Twitter’s real problems. So treat this as noise maybe.

Anyways, ignoring other things unknown and purely focusing on this particular aspect… I thought it’d be interesting to consider what we might gain, if we’re just willing to give up certain things?

So.. assuming static resources ain’t a problem, lets try to have more of those! A single person’s stream (without friends) looks cachable - like a blog that generates html instead of serving dynamic. A single person’s social graph (following who) looks cachable as well. If assumptions are wrong here, then exit(1); So a person’s homepage is his own static stream with static external references to static streams of ppl he follows. Like a html page referencing a bunch of CSS. No server side join or sort. Instead, the browser, with the help of some static javascript, pulls in those resources, does the merging and sorting (and paging) and spit a pretty display similar to the current. i.e. every user’s browser does its own merging and sorting, you know… like, help out will ya?

Would that be better? And what did we give up? The ability to properly browse your own Twitter homepage on a browser without JS & clients using API need a few more pulls and do merging / sorting themselves too.

Can we live with that?

Update: Slim pointed to an article in the comments, and “Part III” of that article, Eran had talked abt the same alternative:
if you only ask this once, the server side batch solution will be faster. But when the client starts asking this once a minute or more repeatedly, resources go to waste and scalability is more expensive. link »
Once the server is only serving data with limited scope, usually providing the messages of a single person with the optional perspective of the reader (to enforce access control), scaling becomes a much easier task as data can be segmented easier. link »
- from Hueniverse: Scaling a Microblogging Service - Part III via sharedcopy.com

Update: Then again, you have scoble who follows 30k other people making it impractical for the client to fetch those streams independently… I guess most systems would decide to place limits and “punt” the issue, e.g. you can have at most 5000 friends. And that’s something Twitter is unwilling to let go.

Lightbox, Thickbox, Greybox - Ugh

I dislike the increasing (still?) trend of websites deploying a lightbox (or thickbox) effect. Not so much for the strobing visuals, but more because it is nothing but an irrelevant technique dragged over from the desktop world to the web - just for the sake of familiarity.

Modal dialog boxes are those which temporarily halt the program in the sense that the user cannot continue until the dialog has been closed link »

Modal dialogs are generally regarded as bad design solutions by usability practitioners, since they are prone to produce mode errors . link »

- from Dialog box - Wikipedia, the free encyclopedia via sharedcopy.com

The web is free flowing and has no vertical constraint. And it’s good that way!

My personal preference as a developer is to use div-expansion/collapse everywhere possible, until I can’t. By introducing new content in-place, and reflowing the rest of the document, a user is not required to take action just to back out. If he choose to proceed, he can interact with the new content (usually a form), or simply continue browsing, scrolling down or click elsewhere.

Even if situation calls for a lightbox-ish design, it isn’t too much to ask for [as much of the] surrounding content to be pushed aside whilst showing a thickbox at the centre of the screen. You get to reduce overlap & maintain document flow at the same time.

Try it, click on the Wikipedia quote above. Though the right columns of this blog is obscured by virtue of their floats, the content of this entry will remain in full view.

"Go away or I will replace you with a very small shell script" v2


Credit: Philip M. Parker
Parker shows a book being created.

Using a little bit of artificial intelligence, a computer program has been created that mimics the thought process of someone who would be responsible for doing such a study link »

It will then open a Word document and export the information into Word, just like a real author would out of their minds, so to speak, or spreadsheets link »

- from He wrote 200,000 books (but computers did some of the work) | Tech News on ZDNet via sharedcopy.com

Reminds me of this t-shirt.

Poor Man's Trends

I’ve just released my Poorman’s Trends Rails plugin as an excuse to do something with my GitHub account. Much like its distant cousin, Poor Man’s CRM, this software aims to be plug-and-play, self-sufficient and useful enough to make sense of existing data.

3 Steps to get it:

  1. Install into vendor/plugins with
    ./script/plugin install git://github.com/choonkeat/poormans-trends.git
  2. Add 1 line into your existing controller, e.g. admin_controller.rb
    include Poormans::Trends
  3. Point your browser to http://yourserver/admin/poormans_trends and Viola!

You should see a simple page greeting you, like in this screenshot:

Poorman's Trends Homepage

Pick a Model class, and you should see some pretty graphs making their AJAXy entrance:

Poorman's Trends Example3
Poorman's Trends Example1

What’s happening is that a few interesting columns are picked up from the Model’s table and a few popular values are picked up from those columns. A count is done for all of them, divided up by weeks and spitted out as HTML tables. These tables are then turned into pretty charts using the nice HTML-table-to-canvas javascript library from Filament Group.

Without configuration nor knowledge of what your application does, is there any hope that it can make sense?

That’s where the "picking of interesting columns" come in. Assuming zero domain knowledge of your application, I’d say the foreign keys are most interesting (in Rails convention, any column ending with "_id"), together with sub-classes ("type" and any columns ending with "_type"). The time aspect relies on the Rails convention of using "created_at".

The end result actually looks pretty good for something requiring no configuration! :-)

Ruby on Rails - Sharing Session for Singapore PHP User Group

Here are my slides to yesterday’s talk. Thanks to all who attended, and Michael Cheng and gang for organizing it! I tried to make it more constructive instead of anything X vs Y

No Template is the Best Template

Update 28 Sept 2008: checkout the working code at github repository, and a short video demo on vimeo.

I’ve been swimming in jQuery-bliss for a few months now - thanks Divya & Tien Dung for introducing me to it! I’m yearning to do it on the server-side.

But to enjoy the jQuery-bliss (aka ”no special case pattern”), I’d need to be in a problem where I have a document to sculpt. My recent toys (cleartz and tryanslator.. and actually most Javascript apps?) for example, are built the via sculpting way: a very basic HTML structure, then Javascript comes in to flesh out its form and introduce function. A very pleasant development experience overall.

Aside: Back in Java days, I remember I used to hate XML DOM parsing a lot. NodeList? hasChildNodes? But XPath came and suddenly pain became pleasure.

On the server-side, Model (of MVC) stuff seems to fit (maybe sqlQuery( "people[gender='Male']" ).update({ salutation: "mister" }) ?) but ActiveRecord (& Ambition?) is already nice to work with, and I have no major problems there.

My gripe is with the View (and consequently the Controller). Views are generally powered by some templating system. These gnarly things are just a pile of bastardized HTML files good for neither designers nor developers. The best templating systems are only best because you’ll choke yourself slower by using them. Anyways, pretend you agree that templates == bad.

Wouldn’t it be nice if your web framework 1) loads up pure HTML as a document object, 2) you modify them in the jQuery-bliss way in your Controller, then finally 3) the resulting HTML is spit back to the client browser?

No nasty <? %> } or custom syntax for dynamic content, or loops, or if-else. Just pure HTML with dummy data baked in.

# display_user.html
<span class="tel">
<span class="type">home</span>:
<span class="value">+1.415.555.1212</span>

Then modify them before serving:

# inside controller
def display_user(doc)
  user = User.find( params[:user_id] ) 
  (doc/".tel .value").inner_html = user.phone_number
  return doc.to_html

And because it is an in-memory tree instead of some opaque String, you can do everything programatically around the document: looping, if-else, removing nodes, replacing sub menus, partials? Furthermore, when you need to do dynamic behaviors on the client-side, the programming paradigm is exactly the same!

Designers can be happy churning out the HTML/CSS in whatever tool they like (be it Dreamweaver or notepad.exe), and developers just poke their DOM.

What say you?

PS: Oh well, I guess what I’m trying to say is maybe we should not waste time finding better ways of constructing webpages (i.e. templating systems) when we could approach the problem as “modifying a document” - which is much cleaner and (with the right API) much easier.