YUI Uploader Implementation

Image Uploading using YUI Uploader Widget

A few days ago the YUI team released the latest and greatest version of the YUI Library, YUI 2.5.0. Along with the usual round of bugfixes and speed improvements this release came with several new components. I personally was the most excited about the Uploader component, as the technology that powers it also underpins the upload process on Yahoo! Video. I didn’t write the upload pages but I’m expected to be able to provide bugfixes and enhancements to those pages, so what better way to learn how they work than from the ground up with the Uploader?

Yes, I know there’s quite a few better ways. This was the most fun sounding way. What can I say, I like 4 day projects in my spare time instead of just reading the existing code. Besides, I wanted a better understanding of it from a lower level than just how the upload code on VYC works.

As mentioned in the YUI Blog posting about 2.5.0, the Uploader is what not only powers VYC uploads but Flickr uploads as well. Flickr exposes more of it due to their allowing of multiple files uploaded at once. That’s not really very easy to do on VYC due to the much larger file sizes being used. Still, it’s there. I promise.

I’ve had a free-for-all, open to anyone image uploader chilling out for quite some time at http://tivac.com/upload/. I wrote it to solve a problem, namely the other image uploading services available at the time sucked. It was just a quick little thing, with some basic JS that would hide the form element and create a new one so you could upload more than one file at a time. When you were done it would barf out thumbs of the images along with some common types of link code for forums and the like. Nothing fancy at all. Well, the new YUI Uploader seemed pretty much tailor-made to work instead of creating new form elements.

Here’s a good representation of my thought process while I contemplated redoing the image uploader.

Method Select multiple files from one dialog? Progress feedback available for updating the UI? Smart enough to not totally hose the browser while uploading? Able to dynamically update UI on completion?
Form + JS No No No Sorta
Uploader + JS Yes Yes Yes Yes

I think that paints a pretty compelling picture of why I’d go with the YUI lib over the original solution. Aside from requiring Flash 9 and not working on the latest OSX because Adobe and Apple are feuding, there’s really no downside. It’s provably better in every way. Since this is just a junky little personal project and not something important, I don’t even provide a fallback HTML form any more. That’s just how I roll.

Actual implementation was pretty basic. The YUI documentation is excellent as always. It more or less started life out as a copy of their Simple Upload Example and then I hacked the crap out of it. The flow ended up being something along the lines of the following.

  1. Browse for image files.
    upload step 1
  2. If you selected one you don’t actually want to upload, clicking it’s filename will remove it.
    upload step 2
  3. Upload the files, watch all the fun progress bars whiz around.
    upload step 3
  4. As each file completes, its filename is replaced with a thumbnail and various different pre-filled link codes for HTML and forums.
    upload step 4
  5. You can also hit the Export button to just get a list of the URLs of the uploaded images.
    upload step 5

It sounds complicated because I’m an engineer and don’t explain things well. The cool nerdy thing about this is how very event-driven it is. For example, take a look at this code chunk.

 

//certain things can only happen once the SWF is ready to rock
this.uploader.addListener('contentReady',   this.swfReady);
//event handlers
this.uploader.addListener('fileSelect',         this.onFileSelect);
this.uploader.addListener('uploadStart',        this.onUploadStart);
this.uploader.addListener('uploadError',        this.onUploadError);
this.uploader.addListener('uploadProgress',     this.onUploadProgress);
this.uploader.addListener('uploadCancel',       this.onUploadCancel);
this.uploader.addListener('uploadComplete',     this.onUploadComplete);
this.uploader.addListener('uploadCompleteData', this.onUploadCompleteData);

There’s an event fired by the flash object for pretty much everything you could want. The only event I found myself wishing it supported was a “allUploadsComplete” method. As it was I had to create a cache of all the file ids being uploaded, and in my handler for uploadComplete I would remove each id from the cache. If there’s no more files left to upload, hey presto we’re done! A little roundabout, especially when having to watch out for files that were selected and then removed.

The other fun thing I decided upon was that instead of using innerHTML and getting back big chunks of HTML from the server (wrapped in JSON, of course) I’d instead use skeleton structures hidden in the DOM. A quick

YUD.get('upload_skeleton').cloneNode(true);

and you’ve got yourself a nice chunk of HTML just waiting for some delicious data to be inserted. The actual insertion is pretty boring DOM traversals, but by using a skeleton as the base it avoids heavy node creation/appending or a massive innerHTML dump. The page’s DOM is a little heavier because it contains these stubs, but there’s only two of them and page weight isn’t a big concern for this project.

The progress meters are basically a total ripoff of Flickr, I feel a little bad about it but I really liked their approach of using a background image and just changing the x offset each time uploadProgress was called. It’s so simple and to the point that I didn’t see any reason to make it more complicated.

So that all works fine, but I was worried that if I didn’t copy the URLs correctly right away the images would be lost forever. That meant that I had to write a little PHP script that would output linked thumbs of all the images that had been uploaded. Since this is a totally unrestricted image uploader there’s been a few instances of people uploading porn and the like. I’d rather not deal with that, so there’s a little delete script as well. It’s protected by a .htaccess/.htpasswd basic auth setup. Nothing fancy, but it keeps other people from being able to delete my images.

image browser

The images are laid out in a big floating grid using Hedger’s very awesome work on getting display: inline-block to work across browsers. Item List Grid : Practice with display:inline block across browsers. It’s a technique we also used extensively on VYC so it was the first thing that came to mind when just floating a bunch of images that weren’t the same height went all crazy.

Oh, right. All the icons are the insanely awesome work of Mark James, specifically his Silk set of icons. I’m sure you’ve seen them used pretty much everywhere. There’s a good reason for that, they rock.

I’ve zipped up the source for anyone who wants it, it’s all pretty well-commented. You will have to handle the file permissions yourself though. UPDATE: Sitzmar wanted me to point out that all the paths are hard-coded for my uploader.  He says I should be ashamed, I say he should shut up.  Also I don’t think I included the .htpasswd (obviously) or the .htaccess.  Those are easily set up, a quick search on the internet will answer all questions.

upload.zip

Sites I read that are worth checking out

This is gonna be a long one, so watch out. I’ve been collecting feeds of people who I consider to have really worthwhile things to say about my chosen career (web development, for the people who don’t know that about me already). I’ve got quite a list, and since I’m of the opinion that sharing is caring I will be providing a huge list of links to sites that I think are awesome and deserve your attention.

I make no claims that this list is authoritative, in fact I’d love feedback pointing out really smart people I missed. I read a lot of feeds but none are more dear to me than the ones contained with my “development” folder. Adding new items to it is a joy each and every time.

  • Usability/Design

    • 456 Berea Street, full text rss icon

      Roger Johansson does a great job of covering usability issues on the web.

    • 90 Percent of Everything: Usability Blog rss icon

      Harry Brignull and Andy Baker cover usability design both on the web and in client apps.

    • flow|state rss icon

      Usability fascinates me, and this is one of the best usability blogs I’ve ever seen. Updates are infrequent, but always incredibly insightful.

    • Looks Good Works Well rss icon

      Bill Scott left Yahoo! for Netflix but we won’t hold it against him. His UI design comments are fantastic.

    • Theresaneil’s Weblog rss icon

      Another really fantastic UI blog. I wish there were a lot more done this well.

  • Web Development

    • A List Apart rss icon

      Great writers and a ton of content makes this a great resource. Doesn’t update that often, but always great content.

    • Bite Size Standards rss icon

      Tiny little nuggets of web knowledge, updates infrequently but with useful info.

    • Ajaxian rss icon

      Ajaxian’s great about updating with the newest and greatest javascript libraries and techniques.

    • Bartelme Design – Journal rss icon

      Small Design firm based out of Austria that posts nice wallpapers and some decent articles every now and then.

    • CSS Help Pile rss icon

      Monster collection of CSS tips and tricks, always a nice reference to have around.

    • CSS, JavaScript and XHTML Explained rss icon

      Browser quirks, CSS techniques, all sorts of stuff. Updated rarely.

    • DOM Scripting Task Force rss icon

      Unofficial group dedicated to promoting good Javascript, I can get behind that goal.

    • Douglas Crockford’s The Department of Style rss icon

      Crockford discovered JSON and writes great articles about Javascript best practices. Also see javascript.crockford.com and JSLint.com.

    • Dustin Diaz rss icon

      A Googler (we won’t hold it against him) yet he still loves YUI (yay!). Produces a lot of really interesting code snippets.

    • Fiftyfoureleven.com rss icon

      Mike Papageorge runs this general web developer resource/weblog. Good stuff though getting a little stale.

    • Firebug - Web Development Evolved rss icon

      Firebug development blog, updates almost never but good to have a subscription to in case it does.

    • Foo Hack rss icon

      Isaac Schlueter is a fellow Yahoo! and writes some fantastic pieces about web development.

    • Garrett Dimon rss icon

      Garrett Dimon’s weblog is wonderfully minimal, yet still provides really great insights into the development process.

    • HedgerWow`s Blog rss icon

      I worked with Hedger last summer, he’s a true mad scientist in the world of web developers. Seriously, I have no idea how he comes up with most of this but it’s insane and amazing.

    • If..Else Log rss icon

      Only occasionally development related, but still a good read.

    • JoeHewitt.com rss icon

      Joe created Firebug, so you know he’s pretty damn smart. Too bad his site appears to be unloved these days. Staying subscribed just in case!

    • John Resig rss icon

      Author of JQuery and an all-around Javascript genius, I love reading about John’s continuing adventures.

    • Johnnie Manzari rss icon

      Interesting observations about the web, definitely worth checking out.

    • Julien Lecomte’s Blog rss icon

      Julien’s the author of the YUI compressor and consistenly posts things you should be reading.

    • Matt Snider JavaScript Resource rss icon

      Matt Snider does a great job of breaking down JS and investigating all the 8 bajillion frameworks floating around these days.

    • Nate Koechley’s Blog rss icon

      One of the main YUI guys, Nate’s super smart and consistently posts interesting stuff. Not always web dev related, but always worth checking out.

    • Particletree RSS Digest rss icon

      Great articles, wish they’d start updating again a bit more regularly.

    • Pete Freitag’s Homepage rss icon

      Pete’s a Cold Fusion guy but still posts things that apply to web dev in general.

    • Ryan on WordPress rss icon

      I like WordPress an awful lot, so keeping tabs on its development is a good idea.

    • Schillmania: DHTML and other client-side experiments rss icon

      Scott Schiller is a web dev genius. I have no idea how he’s so good at what he does, I wish I did. He’s at Flickr now, before that he worked on the really impressive new Y! Photos that was canned.

    • Simon Willison’s Weblog Entries rss icon

      Django co-creator, Simon also writes a lot about OpenID.

    • SitePoint.com rss icon

      Lots of good articles to be found here.

    • snook.ca rss icon

      Lots of PHP talk, it’s good stuff.

    • Solution Watch rss icon

      Features new websites that do things well. Good for keeping up with the current state of web dev.

    • Vitamin Master Feed rss icon

      Vitamin is kind of like A List Apart, another big group of contributors writing amazing articles.

    • WordPress Development Blog rss icon

      Keeping tabs on WP development, even if I don’t have my hands into the guts as much as I used to.

  • General Development

    • Coding Horror rss icon

      Jeff Atwood’s posts are always well thought out and engaging, I love his blog.

    • Joel on Software rss icon

      Joel Spolsky writes about software development from a very pratical place, having done quite a lot of it himself.

    • Programmer’s Notepad rss icon

      Programmer’s Notepad rocks my world. I use it every day and love it to pieces. Keeping up with releases via a feed is tops.

    • Rasmus’ Toys Page rss icon

      Rasmus is really, really smart. Not surprising, given that he wrote a bunch of PHP (the language itself, not just PHP code). He’s another Yahoo! as well.

    • StickBlog rss icon

      Lots of good development tips to be found, as well as general programming talk.

    • untidy blog » PN rss icon

      Programmer’s Notepad author’s personal blog. The feed is just for the PN category, because that’s the programming related stuff!

And just in case you made it to the end of that huge list, here’s a link to everything I read on a not-at-all daily basis. It’s a little ridiculous. My Feeds.

Programmer’s Notepad and JSLint

I’ve been using Programmer’s Notepad for some time now as my go-to text editor for code. It’s simple to use and really robust and over all just a really great tool. As the Javascript I write has gotten more and more complicated, I’ve found myself often hitting up Douglas Crockford’s JS Lint to sanity check what I’m doing. When I found out that Textmate offered JSLint integration via a bundle I have to say I was pretty jealous.

Then I stumbled across a post from Simon Steele, the developer of Programmer’s Notepad.

Tools I Rely On - Those I Use From PN

So now I knew that you could use JSLint from within Programmer’s Notepad, and had a starting point in the link he provides to the Windows Scripting Host compatible version. I began stumbling around the tools menu in PN, trying to figure out how to get the jslint.js file I had downloaded to be usable as a tool. I figured it out with some help from Simon on the PN forums, Issues with using JSLint as a tool.

With that in place, I was able to get JSLint working as a easily-called tool within PN. With a simple keystroke I can sanity check my JS for all sorts of nasty behaviors, it’s totally awesome. Here’s a quick walkthrough I wrote up after mentioning this on a Yahoo! internal mailing list and getting a question about it.

  1. Downloaded the WSH version of JSLint from http://jslint.com/wsh/index.html and copied it into a tools subdirectory in the Programmer’s Notepad directory (just for ease of referencing).
  2. Added a new tool for Javascript files:
    Tools -> Add Tools -> Scheme: Javascript -> Add
  3. Settings are as follows:
    JSLint Settings

Those settings allow for double-clicking the JSLint output line and having PN jump to it, which is really handy. Unfortunately it stops after every error and always complains about a null character at the end of files, but those are minor annoyances. It works really well and is pretty quick to get started, which I appreciate.

I’d love to get it so that it’ll run JSLint every time I save a .js file, but I don’t think PN supports that for tools yet. Something to ask about on their forums, I suppose.

Work Work Work

I am insanely busy with work stuff, but we’re in the home stretch and that’s really exciting. I will share more about what I’ve been doing soon!

Javascript, ActiveX and IFrames

Been working on a project that involves getting some JS to interact with an ActiveX control and then have that work with an IFrame for asynchronous uploading.  It’s been interesting, at least.  Getting the AX object into the page so that my JS could actually reference its methods was the first hurdle.  After a day of banging my head against the wall I stumbled onto a site that recommended using innerHTML to write it out.  Wonder of wonders, it worked great.

I started building up the JS scaffolding  around that and promptly ran into another issue.  While the ActiveX object is thinking it calls a predetermined callback method in the JS to give status updates.  Unfortunately, you can’t declare this callback until the object is in the page.  This means dynamically inserting a new JS file in the initialization after dynamically inserting the ActiveX object.  That’s a lot of dynamic inserting and it makes me a bit nervous but it works fine in IE and since we’re using ActiveX that’s all I have to test!

The next step is getting asynchronous uploading using an IFrame working.  This is being complicated by most tutorials that cover this technique being old as dirt.  We’re talking 2002 old as dirt here, before AJAX was anything more than a cleaning product.  “Script Remoting” is not what I want to do (I have XMLHTTPRequest for that, thank god) but that’s what most of these tutorial focus on.  It makes parsing out the useful info needlessly complex and it’s really driving me up the wall.

Having a a good time at JumpCut though. :D

YUI is so awesome even Google uses it

I was linked by a friend to a very interesting article by Zach Leatherman about the CSS used on Google’s customizable homepage, http://google.com/ig/.

Turns out, they’re using the YUI grids component to space everything out on the page. That’d normally be fine, but the terms of the BSD license for the YUI modules requires that you leave in the copyright notice. Google didn’t do this, so technically they are stealing the code. Tsk tsk.

Here is Zach’s much more in-depth article: Google Using YUI Grids CSS

I can talk about what I did last summer

Last summer I got an amazing opportunity to intern at Yahoo! for 3 months. It was a really cool experience and I learned a ton while working on a project that at the time I could not talk about. As it turns out, that project has finally gone public beta and I can talk about it (at least so far as it exists and stuff I guess).

All New My Yahoo

All New My Yahoo!

I worked on the team that made that, albeit in a earlier and less polished form. It’s hard to tell what of my code survived since according to my old manager they had to throw out a lot of the old designs. However, the weather widget still looks like the same one I wrote with some moderate refactoring. That was cool to see as I’m pretty proud of that weather widget. The “Set My Yahoo! as your Home Page” link is still there and appears to still work the same way. I can’t say for sure on any of this because untangling jsmin’d javascript is no fun.

The new beta can be accessed by going to http://cm.my.yahoo.com/upgrade and I have to say I like it quite a bit. The inclusion of the big ad on the right is too bad but with AdBlock I never see it so it’s a non-issue for me. The team has done a really amazing job on this in the 6 months I’ve been gone, a lot of the early designs have been polished until they practically shine. It’s definitely still feeling the performance blues but I have no doubt that they’ll get it running smoothly before launch.

Implementation Showdown: Y! BBAuth vs OpenID

As some of you may have noticed I was really excited when Yahoo! released their BBAuth tool right before their Open HackDay (My post about BBAuth). It seemed like a great idea, as tons of people already have Y! accounts. Little did I know that BBAuth wasn’t really a good solution as a single sign-on utility. It sounds silly for me to say that especially when the Official BBAuth Site says “BBAuth also offers a Single Sign-On (SSO) facility so that existing Yahoo! users can use your services without having to complete yet another registration process.” In the real world when I implemented Y! BBAuth as a SSO provider I ran into some issues.

First though, a quick overview of how BBAuth works. You set up an entry in the BBAuth DB for whatever site/webapp you’re going to use it with, providing some general info like contact details and the endpoint for when a user has successfully logged-in. Once that’s done you get a shared secret and an application ID. These are used in all future calls to hash info so that it’s reasonably secure and can’t be easily snooped along the wire. A special login URL is crafted that points the user to Y!’s login page, and once they enter their Y! credentials they are asked to give permission to whatever site for the info that has been requested. Once they agree they are redirected back to the site along with some encrypted info depending on what the site had asked for.

BBAuth works just fine, but even if you don’t ask for any Y! info your users still have to go through and agree to allow the site to use their info. My problem is then that I don’t want the user’s info. When I set up my site to use BBAuth I even told Y! that I didn’t want to use any info. All I want is Yahoo!’s promise that the user is a real Y! user and some sort of unique hash that my system can use. I’ll get the info I need for whatever app I’m running at the time, but none of what I do would be asking for Y! Photos info. I’m not saving any login credentials via cookies or anything, I just want a simple login mechanism that I don’t have to maintain. Clicking through one page doesn’t sound so bad until you realize that every time the user logs in they have to go through it. That’s pretty annoying and makes the jump to a Y! login page even more jarring.

There’s a post on the ydn-auth group by Jeremy Zawodny (Jeremy’s Post) that addresses this issue and says that this functionality is apparently a special case for BBAuth. Seems a little odd but I suppose that they were more concentrated on the usage cases for sharing a user’s Y! info with a 3rd party site. So they know about it and are looking into it, but until then I can’t really say I’m pleased with the actual implementation of BBAuth as a SSO provider.

After implementing BBAuth and being disappointed with the SSO performance, I began looking into other solutions. I really didn’t want to require users to sign up for yet another account because everyone hates that. It seems like there’s been a lot of buzz lately around OpenID (Technorati stats for “OpenID”). I figured I should at least give it a shot considering the large surge of interest behind it. First I had to figure out how OpenID actually worked.

Short Version: instead of a login that is some clever play on your name (Tivac lol) or something you identify yourself using a URI that links back to an identity provider. The identity provider stores your details and provides password authentication, so that when you visit an OpenID-enabled site you enter in your identifying URI and get bumped to your identity provider. You log in there, specify what details the site should be able to see from your profile, and then say “Ok” and get sent back. Note that this is what it’s like using MyOpenID.com because I was too lazy to set up my own half-assed identity provider.

So, how does OpenID work as a SSO provider? From a user’s point of view it’s pretty dead-simple. Looking at it as a developer adding it to an existing property proved to be considerably more work than I expected. I used the JanRain OpenID Library for PHP when working to get OpenID logins enabled on my testing site. Now I know the title of the article doesn’t say I’m comparing the libraries I used for each, but it’s still interesting to look at the differing amounts of code required for BBAuth and OpenID implementation in a fairly typical hosting environment.

The first thing that struck me as kind of odd was that I needed to move two folders totaling 33 files onto my shared path. I had forgotten that OpenID uses Yadis (Wikipedia page linked because the homepage at yadis.org has been vandalized) as part of the protocol. I didn’t realize that I’d need to be installing classes to support Yadis as well as OpenID. Once I had gotten all those files copied into the proper places and confirmed they were on my shared path it was time to look at the examples provided.

OpenID provides a decent example setup for both consumer and server roles, where consumer is providing the SSO login and server is actually running your own identity server. I only want to run a consumer right now, so I used just those files. After getting the form setup and trying it out I discovered that my PHP install has no big integer math library installed. This necessitates running in “Dumb Mode” which according to the documentation means my login form is more susceptible to relay attacks. Great.

After defining a new constant telling OpenID to run in dumb mode I was able to be successfully bounced out to my Identity Provider, sign in, and be bounced back to my site.

Avoiding any in-depth analysis of the security between the two as I’m not qualified to comment, which did I prefer implementing and using? That’s not actually an easy question to answer. BBAuth was certainly much smaller in terms of code required. It also has the advantage of using Yahoo!’s login system, it’s not exactly scientific but I don’t know anyone who doesn’t have a Y! account. So BBAuth has a huge built-in userbase right there. Both services require bumping the user out to a 3rd party login page so that’s a draw. I knew it was part of how both worked going in and it isn’t an issue for me in this case. Implementing OpenID had some bumps but thanks to the distributed nature of the project I don’t need to go sign my web application up anywhere. If I want to shift around page names I don’t have to remember to go update any entries in a Y! database anywhere. Any OpenID Identity Provider worth their salt will also provide a mechanism to say “Always Allow this site” which is something that BBAuth currently lacks.

In the end I wasn’t able to come up with a conclusive win on either one. BBAuth has that annoying extra page every visit but tons of people have Y! accounts. Implementation was also pretty quick and easy. OpenID took more work but with a decent Identity Provider will only require one page not on my site. It’s also got the whole distributed thing going for it. As it stands now I’ve implemented both and users can just choose their favorite. I’d prefer to just offer one but I don’t think there is one that offers a compelling enough featureset over the other to go with a single one yet. Going with both gives me flexibility and an excuse to keep playing with both of them.

Yahoo! hosting YUI Files

They just announced this on the YUI Blog, Yahoo! is allowing anyone and everyone to use their nicely cached and gzipped copies of the YUI libraries. This is awesome news, as it means I no longer have to manually update 3 sites every time a new YUI release comes out.

http://yuiblog.com/blog/2007/02/22/free-yui-hosting/

The actual article on how to use the hosting is located at http://developer.yahoo.com/yui/articles/hosting/