Dawn.com has launched their beta website available via dawn.net. Going with a phased roll-out through Beta is the only thing they seem to have gotten right and one can only hope they address the issues with the website or go for a complete overhaul. After spending 15 minutes on the website, I only hope they don't take this live any time soon. Following are some things that immediately stuck out like a sore thumb.

1. The unwelcoming invitation for dawn.com Beta is like the annoying popup ads you get on cheap warez sites. They could have at least tucked it in a corner instead of having it overlap navigation and content. And whatever happened to using cookies and giving users a "Do not ask me again" option. It's an extra ~20 lines of code in the absence of which this annoying popup is displayed to the many thousands on every visit, every day.
   
2. The website is optimized for 800x600. This is 2008. Only 8% of users use that resolution. 86% use 1024x768 or higher which provides 30% more screen real estate. All that space can be put to good use. This is a common problem with developers here who have very little interest in the discipline and read little or no literature pertaining to their field. If nothing else, I would suggest that the Dawn.net team take some lessons from news.bbc.co.uk or iht.com and follow their design principles, if they have any principles to begin with.
3. The URLs are completely meaningless. They could have slapped together a few regular expressions with mod_rewrite or equivalent and respected the URL as a UI. Even if they fix this now, the older links will become obsolete and result in all existing links to those stories to break, a common problem known as link rot. Why downgrade from existing dawn.com/2008/02/28 type URLs?
4. Placing advertisements in the main navigation is not only annoying it is a negative user experience. If they understood point (2) they would not be struggling to squeeze in these ads. Worse still, the advertisements cause navigation to drop below the page fold (the area which is not immediately visible and can only be reached by scrolling). That's a usability blunder and shows that users are not even a priority. A side by side comparison of BBC's and Dawn's menus clearly highlights this intellectual divide:
   
5. There are other design, usability and information architecture issues but I want to keep this post short and avoid a lecture on design and usability so I'll say a little about their code. I was hoping to find half decent code because it's such a high profile project but the code is atrocious and even comic relief doesn't help (skyscraper spelt skycrapper in the code comments). Just consider the following code pattern that occurs close to a dozen times:
   PLAIN TEXT
   HTML:

   1. <script type="text/javascript"><!--
   2. var x = 0;
   3. var feaz = 0;
   4. // --></script>
   5. <script type="text/javascript"><!--
   6. feaz ++;
   7. x ++ ;
   8. switch (x)
   9. {
   10.   case 1:
   11.     document.write('<td width="280" align="left" valign="top" class="fbg">...</td>')
   12.     break;
   13.  
   14.   case 2:
   15.     document.write('<td width="280" align="left" valign="top" class="fbg">...</td>')
   16.     break;
   17.  
   18.   case 3:
   19.     document.write('<td width="280" align="left" valign="top" class="fbg">...</td>')
   20.     break;
   21.  
   22.   case 4:
   23.     document.write('<td width="280" align="left" valign="top" class="fbg">...</td>')
   24.     break;
   25.  
   26. }
   27. // --></script>

   I never came across such code even when I was teaching Introduction to Programming at university. Why would anyone write a switch statement which says if value is 1, print X, if value is 2, print X, if value is 3 print X and if value is 4, print X. Why use a switch statement if all conditions print the same thing? It defies common logic. It's code like this that causes pain down the line and simple feature requests can take days to implement and introduces bugs waiting to go off like land mines.

6. There are no semi-colons to terminate some of those JavaScript statements (line 11) which is standard practise. HTML is being output using JavaScript when just plain HTML output would do (line 11). Unless there are very good reasons, the middle-tier is supposed to handle this sort of logic, not the front-end. The table cell has a CSS class which should be used for styling but the style elements are also defined using inline HTML (line 11). So if you wanted to increase the width of the columns, you must edit both the CSS and the HTML. What then is the purpose of using CSS? It defeats the whole purpose of CSS. The code uses meaningless and arbitrary variable names (lines 2 and 3).
7. If you look at the CSS files (main.css and component.css) it's plain to see that no one taught these developers what the Cascading in Cascading Style Sheets means. I also noticed some redundant CSS. Why write the same code twice? It takes up bandwidth, takes up more time rendering, more time to maintain and introduces bugs (developer updates one snippet but forgets to update the second identical snippet). The main.css is 34KB and the component CSS is 52KB. Then there's some inline CSS. Wow! news.bbc.co.uk has a layout way more intricate and its ~65KB versus Dawn.net's ~90KB of CSS. For example dawn's h5 and h6 elements have the exact same style but instead of defining them using the same CSS block, Dawn.net repeats the 12 lines of code again. Dawn barely stylized its header/footer or menu bar. Even with 90KB of bad CSS, Dawn uses images for section headings instead of styled text. Imagine trying to manage or sift through 5000 lines of CSS code for the most minor aesthetic changes. I am considering using their CSS to exemplify what not to do in any future webdev presentations.

How can one even move to more serious guidelines for web development. I am so befuddled by this code I cringe just thinking about what kind of blunders they have committed in the back-end and middle-tier. This website does not even come up to par with most blogs out there, let alone be worthy of a news website. Even their current dawn.com website is a class-act in comparison and that has been around since at least 1997.

I am guessing this new website is going to be rife with problems going forward given its precarious state. Feature requests will probably take days to implement because of the poor architecture, and cryptic and unmanageable code. The only winners here would probably be the contractors billing Dawn for weeks/months of labour for even the most meagre feature request. And sadly, dawn.net might be in a bit of a sticky situation here with their contractors. If you want to get rich, build something that only you can maintain and takes you days to change so the client is locked in and ends up paying hefty amounts.

Update 8/31/2008: They don't even have spell check in their CMS (Pakistan spelt Paklistan in title). It's hard to image a publishing company using a publishing platform without spell check. Even my blog has spell check.

The recent PASHA Career Expo 2008 provided a great platform to engage with the community. Following are slides from my presentation for Developing for the Web.

The default WordPress search engine barely qualifies as a search engine. It's results are not that relevant, the search page shows the actual blog entries instead of a list of links of resulting pages and the experience gets worse if the search query gets the least bit creative.

I think everyone should deprecate the WordPress search engine and move to Google Custom Search. There is simply no comparison. You can see the search results for AJAX on my website.

How-to for Google Custom Search Implementation in WordPress

Having recently implemented it on my site I don't think it gets much better than this. Implementing it in WordPress is trivial and the results are of the highest quality.

• Head on over to http://www.google.com/cse/ and register for an account. Follow the prompts all the way until asked to select the kind of layout/code options.
• The best Google Custom Search configuration I think is to select the iframe option with advertisements placed on the right. Secondly, you will want to specify the search results url as something like http://yourdomain.com/search.php. More on that in a minute.
  
• Subsequently you will be shown two textarea's. The first one is the search box code which shows the input field where queries are entered. You can paste this code in your header or index template (header.tpl or index.tpl in the wp-content/theme/ directory) depending on where you want the box to show up.
• The second code box is for the search results page. To display the Google Custom Search results on a fresh page, create a file called search.php in the root of your web directory (this corresponds to the results url you entered above). Within this page you should paste both code snippets. The Google Custom Search box code will display the search box to the users once again so they can see their own query. The Custom Search results code will display the actual results below the box. This is what your search.php will look like:
  PLAIN TEXT
  PHP:

  1. <?php
  2. // these two lines get the theme page initialization code and header
  3. require('./wp-blog-header.php');
  4. get_header();
  5. ?>
  6.  
  7. <!-- Google CSE Search Box Begins  -->
  8. ...
  9. <!-- Google CSE Search Box Ends -->
  10.  
  11. <!-- Google Search Result Snippet Begins -->
  12. ...
  13. <!-- Google Search Result Snippet Ends -->
  14.  
  15. <?php get_footer(); ?>

• That's it, you're done.

Your page won't be instantly indexed and may take a day or two, but that's a small price to pay. And if you show your 5-10 most recent blogs in the sidebar then you're good anyway because users can navigate directly.

A while back I started what I thought would be an easy little project for an Online Scrabble Game clone. It's now completed and launched as Beta. I am calling it Yabble because the other interesting *abble words are quite common on the web. My reason being that instead of bookmarking sites or remembering a URL I often only remember a keyword that will help me rediscover the website. With Yabble it's easier to get into Google's top 10 as compared to Dabble or Gabble which return way too many results.

I do not want to get into the design details in this post except to say that the game is written using PHP and AJAX and uses XML files to store game state on the backend. I also used the Prototype JavaScript framework and Leightbox.

I used YUI Compressor to minify the Javascript and CSS and get some code obfuscation as a bonus. If it demands enough attention, I'll take time out to add more features and refactor some code and design.

The group I am working in at Microsoft has to do with business intelligence and advertising displays--this is a very interesting space to be in at the moment as it is central to the monetization strategy for the MSN division (now Windows Live). It's something that has been forced upon all Internet players.

In the pre-Google era, advertising-based services were a failed model--to use the then oft-quoted quip "there is no such thing as a free lunch". That changed when Google proved that ad-based services can make money and they did this through state-of-the-art datamining and simple two line text-based ads which marked a complete shift in the conventional thinking. It was very innovative and it undercut premium services like Hotmail which eventually followed suit. It is the driving force behind Google's billions (2.7 billion in revenue for their most recent quarter).

Recently I got a an internal newsletter inside which I stumbled upon a few Microsoft bloggers in this space and spent a bit of time reading through some interesting posts. What prompted me to write this post was Mark Jacobson's point regarding user profiling and how Google lacks demographic data on its users. You have to ask yourself, why is it that Google is not interested in this information and how much of a disadvantage do they have? Do they know something we don't?

If you sign up for Hotmail or any of the Passport services, one of the forms you encounter will require your birth date, gender and location. Google on the other hand doesn't ask you for any of this information--their form is extremely minimalistic in that regard.

Why Demographic Data

Most conventional marketing and research is based on targeting demographic profiles. If you are a 29 year old male living in the 90210 zip code (Beverly Hills) it would indicate that you are in the 1MM+ income group. That helps in targeting ads to you. Your very first ad could be for a new Porsche or a Versace suit. This is traditional marketing and while I don't know how optimized those heuristics are, but on the web you can--and should--leverage a lot of the newer metrics it has to offer.

As an aside, ads on television networks also have a lot of room for improvement in the ad relevance arena as we step into the digital age, however, there are technical hurdles (harder to datamine videos; harder to show different ads to different people).

Context is Critical

So I am a 19 year old whose zip code is in the university district and you can show me ads about a Dave Matthews concert in the area but how likely am I to click it based on my age and zip code? The probability that I am a student is pretty high because I'm the right age and in the right zip code but that also assumes you have prior research to deduce user profiles for that demographic data. And now you still need to know if I like rock, heavy metal or house music. You also don't know if at that very moment I am very busy researching symptoms of an illness or researching my next big investment move in a particular stock and don't give a hoot about some concert.

On the web you are what you read. It's about contextualization. What should matter to advertisers is what I am looking at, at any particular moment because that is what is most relevant to my interests at that very moment. With demographic data you are targeting most 21 year old students for most of the concerts most of the time (up until the concert is sold out or the marketing campaign ends). All the while you are hoping to cast the biggest net possible to grab a tiny audience of concert-goers in that region. There is a lot of wasted effort. Most of the people don't care about the concert. The clickthrough rates will be very low. In fact, this model was so bad that nearly all services I can think of that were built around ads failed with this business model around the dot com glory days.

If you shift your focus to contextualization for achieving higher clickthrough, you get a lot more out. I am on a page looking at the discography of U2 and you show me an advertisement to purchase the U2 iPod Nano or a U2 CD. Those ads are agnostic to demographic data. This means that you cannot show me an ad for a U2 concert in my state because you don't know my zip code. However, as an advertising company you can only so show many ads on any page and the U2 concert ad is better left for the Seattle radio station website where locals visit. Local ads on localized content, adult ads on adults content, children's ads on children's content, rich ads on luxury content and so on.

When you have good contextualization, the value of demographic data falls considerably. That said, demographic data obviously has its uses. A page like msn.com which is generic does not represent the mood of the reader since it's dynamic and very general. On a page like that, knowing if your user is male or female helps promote the right kind of ads.

AJAX in itself is a very simple technology, especially with the availability of the right tools. You make a request and pass some parameters, get a response and you render appropriately. The only thing is that the entire process is tedious and if you are the kind of person that likes to do their own plumbing, things can take time. I recently started a web 2.0 project and am nearing completion of the login page. If you are at the beginning of the web2.0 learning curve (barely familiar with JSON, REST, and Prototype) you will find this post interesting. Or, if you have ever designed your own login page, you might find some details here quite useful.

Get Equipped

First things first, find the right tools for the job.

You need a cross-browser Javascript framework to overcome the discrepancies in different browsers and simplify your life. You can skip all those AJAX tutorials out there and just take things for granted when you adopt the Prototype framework. It's small and has a clean cut library. Though, I should add, I felt the absence of a cross-browser attachEvent but this here script filled that void. As with all things open source, there are different camps and I considered joining the JQuery camp, however, Prototype has a better roadmap and larger community due to its Ruby on Rails endorsement so I went with that.

Next up is client-side validation to help you conserve server load and precious bandwidth by avoiding trips to the server just to validate the password field is not empty. Andrew Tetlaw's validation.js is built on top of Prototype so it fits the bill quite well. It's small and I really love the way it tackles validation using CSS classes. For example, the validate-email class verifies the field has a valid email address (I was also able to define my own custom validators and error messages very easily):

And what would an AJAX application be if you didn't sprinkle a few animations and effects. I simply wanted to fade text in and out (for usability reasons I will mention later). Be warned though, it is easy to get tempted by all the cool effects and I wouldn't be surprised if all sorts of fancy effects start showing up as these FX libraries trickle down to the lower echelons of web enthusiasts. At first, I went with moo.fx because it's supposed to be superlightweight. However, I ran into some issues with it in IE and switched to Script.aculo.us. Contrary to my initial beliefs, Script.aculo.us is just as small as moo.fx when compressed and possibly better in other arenas.

Speaking of compression, it's probably a good idea to use a reliable JavaScript compression tool before going live with the website. I haven't done this yet and I am certain I'll run into a few other bugs but I'm also certain someone else has already compressed the popular libraries and shared them out. Since all my libraries have relatively big communities, I should be safe here.

Lastly, to exchange data between the web service and the client side Javascript, there are two camps: JSON or XML. I like JSON (JavaScript Object Notation) because it's very lean compared to XML and much faster because it doesn't need any complicated parsing. Instead of using <firstname>aleem</firstname>, JSON simply passes { 'firstname':'aleem }. And then, instead of parsing the XML using XPATH or whatever, JSON simple calls the eval() function which treats the string literal as an object. The only reason to go with XML is if you want to exchange that data with other non-AJAX services. Even so, JSON is well-formed so it's likely you can convert it to XML (there might be tools out there already to accomplish this). Prototype does the client-side parsing and on the server side I chose JSON.NET which takes .NET objects and converts them to JSON for Javascript consumption. It's open source so I can optimize the library and very easily get rid of functionality I don't need.

A Simple Form: Username, Password and Submit

It seems deceptively simple but if you want it done right, there's quite a bit of plumbing to do. Let's begin with the client-side plumbing. Using my validation tools I ran the following client-side validation:

• Username: should be non-empty, minimum of 5 characters, maximum of 15, alphanumeric and must start with an alphabet. The regex looks like this: ^[a-zA-Z][a-zA-Z0-9]{4,14}$
• Password: should be non-empty

Further, every time the user hits the submit button, the error messages fades back in. The reason for doing this is because the user will not know the form has been resubmitted using AJAX so the fade-in provides a visual cue that a new action was performed.

The same validation needs to be repeated on the server side. This redundancy is necessary for obvious security reasons--anyone can bypass the form and submit directly to the server. Assuming the validation goes well, the server can respond in any of the following ways:

• Username does not exist. Display the error on the client side in red and set the focus() to the Login field for convenience.
• Password does not match. Display the error in red, set the focus() to the Password field and select() the password field text. The reason for selecting the password field is so that the user can start re-typing the password right away. This is not necessary for the Login field however, as it is not asterix'd like the password field and can be corrected visually. Also, select() is better than resetting the field since it gives the user the option to deselect and do the correction.

I did run into Javascript quirkiness here. The order in which the event handlers are executed is convoluted and seemingly non-deterministic in IE. I took me time figuring this out and discovering this is a known issue. Remedying it was simple though--I stuck with only one event handler when order of execution was important. This problem came up during form validation--attaching client side validation and server side validation to the form and having them run in order so if the first failed, the latter would not be executed.

The Server Side

On the server side I am using ASP.NET and wrote HTTP handlers using ASHX files. I also turned off AutoEventWireup which is excessive and unnecessary--it automatically wires up event handling functions to the page. The AJAX framework passes requests over HTTP using GET and POST in general. This is the REST approach and if the Wikipedia explanation seems confusing, just ignore it and grasp the following HTTP request:

That's the request that gets sent out when the AJAX framework makes the call to the server. The thing to note is the last line which contains key/value pairs with three keys: r, username, password. When the server gets this, it parses this and makes a call to login("FOO", "BAR") and sends an HTTP Response back:

Ignore the X-Powered-By and other obviously useless headers (I intend to drop them from the server response). When the client-side AJAX framework gets that response it first looks to see if the content contains "true" (last line). If it does then authentication succeeded. For the sake of standardization I will switch to JSON and respond with something like { 'result': true } and then simply use: if(result){...} in my Javascript code. However, I use Enumeration types on the server and JSON.NET does not support serializing Enums to JSON, so there's some work for me here. Anyway, so I grab grab the entire Set-Cookie header and pass it to document.cookie to set the authentication cookie on the client. This cookie is used to verify that the user is authenticated and the login process is now complete.

End of the Beginning

Understanding the guts of the system is not critical to getting the job done but if you want to get intimately familiar with Web 2.0 underpinnings you might want to consider using WFetch or the Live HTTP Headers plugin for Firefox which is terribly useful. The Venkman Javascript Debugger is indispensable if you are into any kind of Javascript development.

I have had a recent inclination toward ASP.NET even though PHP has been very good to me. If you are comfortable with ASP.NET and Visual Studio then you certainly want to use it for learning. PHP will not allow you to set breakpoints, inspect the stack or make runtime modifications. If you are using PHP though, be sure to turn on error reporting if you haven't already.

My login form still needs a "remember me" checkbox and it needs to handle timed out users so after re-login they can be redirected to the page where they timed out. Once the login experience is all squared out, I will move on to designing a quick and simple sign-up experience.

5/29/2007 Update: I wrote Prototype.js does not offer a cross-browser attachEvent, but Event.observe offers just that. For some reason I overlooked it at the time of writing this post.

ASP.NET does not aspire to PHP but it has good reason for envy and the two are undeniably in competition. The ASP.NET platform is quite something when you contrast it against PHP which doesn't even have a development platform worthy of note. It has a clean cut architecture, easily discernible roles for each component and object oriented support. The Visual Studio .NET development environment alone is enough to persuade people to switch to ASP.NET because it's that good--not only for development and debugging but also as a learning tool to understand the flow of the application, inspect the stack and heap, request and response, the current state of objects and other advanced features.

Why then is ASP not giving PHP a good run for its community and user adoption?

Timing

ASP was a late bloomer. PHP was one of my first obsessions with programming (discounting BASIC lessons I took when I was 13 and fell out of a year later because I took to playing games) along with the then rudimentary JavaScript. ASP was no where to be seen primarily because it was not free and endorsed mostly by the professional community. Visual Studio Express attempts to remedy this, though the thought of what could have been if it had been introduced around the time PHP3 was making its rounds would certainly makes some executives flinch.

Platform Strategy

Not all of it was timing. Arguably, even if all the tools made available in Visual Studio Express suite (SQL, Visual Web Developer et al) had been offered earlier, ASP might not have achieved mass user acceptance. A product like WordPress (which powers this site) could never have gained the same level of support if built in ASP.NET. ASP.NET gives protection to the source and is most conducive to building proprietary solutions. You could write a charting library, wrap it up as a DLL and put out a trial version to everyone. Since it runs on the web, enforcing licensing is much easier by having it ping a licensing server. Monetizing it is easy. In PHP this is not the case. The source is there for everyone to read and anyone can and probably will write a free version of any WordPress plugin that you would want to charge for today.

Further, the development ecosystem is scattered with tools and libraries attached to viral licenses that oblige developers to share any derivations arising from these tools. When the ecosystem is built on an open-source, share alike license its difficult to build on the shoulders of the community without giving back to the community. The community has a built-in, self defense mechanism that ensures its survival and results in rapid growth. WordPress and Firefox have quite literally every plugin you could possibly ask for. Installing the plugins is trivial and hacking or extending them isn't too hard if you are a developer.

No comparable WordPress equivalent exists in the ASP.NET world wherein an option is available to keep the source protected and more often than not, the option gets excercised, locking out community engagement in turn. ASP.NET has typically been aimed toward the professional not the amateur though it is now trying to find some middle ground to increase user adoption but I doubt it will ever get there for the fundamentals reasons mentioned here.

Costs

The monetary costs, as well as the cost of adoption are both higher for ASP.NET from my experience. Hosting an ASP.NET application on a Windows Server requires higher licensing costs for the hosting providor which in turn are transferred to the customer, resulting in more expensive hosting for ASP.NET applications.

The cost of learning for ASP.NET is also higher. Visual Studio .NET does provide all the plumbing and template for starting off but the underlying plumbing is quite complex. The ASP.NET page life cycle is not trivial and if you aren't comfortable with a debugger you'll find it harder to experiment with it. PHP's scripting model although crude, allows would-be developers to simply dive in. My first PHP page was as simple as saving the file with a .php extension and wrapping the code in php tags with a print statement. No includes, no headers or strong typing, no nothing. ASP.NET will require building out an entire solution if you follow the prescribed development route and though automatic, it still makes it harder to just dive in.

The lower cost of initial adoption results in a wider user base falling into the PHP funnel.

Communities

PHP has a feedback loop built around good karma. Some random people helped me overcome my hurdles with PHP and in turn I learned to empathize with other new comers and offered my support and experience. I find that ASP.NET does not have the same level of community support which scares some people away and doesn't provide the confidence of crowds. The network effects of a larger community also means that there are more people along each stage of the learning curve. Whether you are just starting off or you are making the switch from PHP4 to PHP5 or you are trying to hack some low-level module, the likelihood of someone else tackling the same problem around the same time is much higher and this kind of peer support is a good motivator.

My Sentiments

I am a long time PHP fan and love the fact that WordPress is a product of it. However, if I were to build a web service or website for my own purposes I would be more inclined to go with ASP.NET. Some of my old ticks are still there and I prefer to do much of the underlying plumbing by hand as I find generic solutions are excessive. I also get a greater understanding and consequently feel more comfortable with the application.

Microsft's partnership with Zend should be a win win for both communities. Imagine loading your PHP solution in VS.NET and having auto-complete--writing WordPress plugins would become a breeze.

I don't really know Scoble but he is terribly popular and he has interfaced with lots of smart people at Microsoft. However, his response to my previous post didn't provide anything substantial to chew on but I'll address it anyway. His response:

you can't live off your existing community and be "cool".

Why not? That's what all companies do. Make money off consumers and get to be cool in the process (iPod, Google, et al).

His reasoning:

Why? Cause if you're a consumer company and you want to see growth, you've gotta be interesting at minimum.

So "you can't live off your existing community and be cool because if you're a consumer company, you've got to be interesting". That reasoning doesn't follow and is incoherent.

Either way, one has to question: Why can't SoapBox or any other video service besides YouTube be interesting? AltaVista was interesting, Netscape was interesting, The Rolling Stones were interesting. This is classic fad mentality and the science of fads is exactly what is feeding this frenzy and clouding judgment. Buying YouTube would not have made Microsoft instantly cool. I wish that were the case. I wish people would stop judging us by the cool things we purchase. That's so superficial.

Some flame bait, and I'm biting:

That big audience that Microsoft has? It's theirs to lose.

That's unwarranted hyperbole. The MSN network has grown consistently and expanded into other services as I pointed out previously.

As a business we have a due diligence to make money. Maybe we got the valuation on YouTube wrong (things of value: denying the competition, acquiring content, creating PR or becoming cool, revenue projections etc). Or maybe we did not want to engage in yet another bidding war knowing Google's ability to better monetize YouTube would've left us in a trailing position. Those are both intelligible arguments and things I am sure we considered.

Again Scoble is widely read and I hope that his post doesn't entice a gross underestimation of the MSN community and Microsoft's position on this. I don't see us backing down and the video market isn't running away. We will channel MSN users toward SoapBox when its ripe for the serving much the same way Google channels its users to their Sidebar, Videos, Froogle etc.

The Microsoft Story

Let me start with my take on Scoble's post that Microsoft should have bought YouTube rather than take the build-it-yourself route. Scoble is saying we should pay 1.65 billion odd dollars for the community, not for the service. Some others are saying Microsoft failed to close the deal. But the truth is that Microsoft didn't really lose out on this deal--it just didn't make business sense at those prices. Steve Berkowitz (Senior VP, Online services) recently sent a mail to employees outlining our position:

...MSN has a huge global audience of 465M users per month that we can tap into and focus on Soapbox to contribute and share videos. We have a far better strategy around integrating Soapbox across MSN, Windows Live and Microsoft services like Windows Mobile, Xbox and Zune, that we couldn't have matched with an acquisition.

We don't need to buy any community, we already have a really large one we can tap into.

Consider how we leveraged the Hotmail community. What we get from Hotmail is far greater than one could attribute to the dollars generated. Hotmail could be attributed as the primary grower of the MSN network. We delivered the Hotmail Notifier that sat in the system tray and evolved into a messaging application and drove Hotmail users to what eventually became MSN Messenger. Messenger drove traffic to MSN Spaces with the gleam feature which helped us dominate the blog market share overnight. Messenger now has built in search that drives traffic to MSN Search and the default install of Messenger is also driving traffic to the IE toolbar, Live Sign-in Assistant etc. All those users originally came from Hotmail--users who wanted to connect.

Microsoft has plenty of users. Building a video service is very easy.

The Value of YouTube to Google

I would've considered the $1.65bn price tag to be exorbitant and reminiscent of the dot com era. But Google found it justifiable.

However, what YouTube means to Google is much more than what it means to Microsoft or Yahoo. The simple reason is that Google has a better ad infrastructure at present which will allow it to better monetize YouTube. They can recover their costs much faster than anyone else out there and so they can afford to pay more. There's also the competitive aspect. Google would rather not see YouTube go to anyone else. YouTube is a big driver for Google's ad traffic and losing it to someone else like Microsoft might mean the loss of all those ad dollars.

Further, Google does not have a large registered user base. Google Search is their primary traffic driver and it's a hit-and-run type of deal. Google could potentially convert YouTube users into registered Google users or channel them to other Google services much like Microsoft did with Hotmail. The reason they will have more success with getting these users to register versus search users is that these users are lurkers. They hang around YouTube spending idle minutes entertaining themselves. Search users on the other hand are task driven--they want to find whatever it is they are looking for and be on with it. This makes YouTube all the more lucrative for Google.

YouTube's Legal Hurdles

The most intriguing aspect regarding this whole ordeal will be the reaction of MPAA and RIAA to YouTube which is basking in the glory of copyright infringing videos. This will be similar to Napster albeit Napster's traffic was almost entirely composed of copyrighted material while YouTube thrives on plenty of legitimate content. Further, if YouTube falls under the "safe harbors" of the Digital Millennium Copyright Act of 1998, the burden of finding copyright material will lie with the copyright holders who will be required to provide a list of copyrighted material to Google to take down. Napster failed here for technical reasons as they could not satisfactorily comply with or facilitate the major copyright holders to weed out infringing materials.

YouTube's story is different as theirs is a much more tractable problem. One idea might be for Google to provide administrative accounts to the large media companies (and complying primarily with the major copyright holders is the key here) to remove infringing content themselves or possibly build a "community moderation" system. The latter will probably not work due to conflict of interest--the community finds value in free copyrighted content but the design could be modified to reward the community of moderators. YouTube could also hire auditors working around the clock to comply with the smaller players though Google is a strong believer of autonomous, scalable systems so its unlikely they will take this route in the longer term.

YouTube also differs from Napster in that their architecture is considerably different. Napster's P2P network meant there were millions of infringers floating millions of copies of any particular song with hundreds of variations. YouTube has a very limited number of duplicates and uploading the same videos under different guises doesn't make sense as the user will not be able to find or share it without the copyright owners being able to do the same. Further YouTube controls the content format which means that copyrighted content cannot be wrapped inside an encrypted layer which is how Napster users were able to share banned songs.

Finally, since the big copyright holders are the ones with which Google will primarily be concerned it will mean that YouTube can do traffic profiling to find videos with massive infringement. It doesn't make sense to go after a copyright video that's being viewed by < 1K users when there are videos out there being viewed by 50K+ users.

For all these reasons and more, Google/YouTube should be able to pull this off. The problem is easily surmountable.

In Summary

Google will weed out the copyrighted content and the problem is not hard to solve. They will get their moneys worth through direct monetization via AdSense and also by channeling these users to other Google services. It did not make sense for Microsoft to pay a similar amount for YouTube's users since it already has a large registered user base from which it can carve out a community of video watchers and it could not have monetized as much from these users. Microsoft will continue to build upon its ground-up service, SoapBox which will be woven with other technologies like Zune, Spaces, Messenger and Mobile.

The legal hurdles should continue to hit the press in the coming weeks/months possibly wavering Google's stock up and down. Bloggers at the same time will continue to feed the frenzy calling Google crazy and brilliant.

It's been a while since my last post partially due to other priorities and a bit of travelling. I have migrated the site to WordPress and away from Movable Type. Primarly the move was because Movable Type being written in Perl is a real turn off--I don't want to touch that code with a 10 foot pole. WordPress is written in PHP which makes it very easy to play around with. Secondly, WordPress is a lot easier to manage and a real relief after Movable Type. Migrating the posts, writing my own theme for the site design, fixing some char encoding issues that were lurking in the posts, adding some plugins for syntax highlighting, fixing some htaccess issues, creating WordPress Pages, updating the BlogRoll and ensuring XHTML validation and other tweaks used up a good chunk of my weekend... which isn't so bad considering the bulk of this time was spent on designing and coding a new theme for the website.

One hiccup: some of the older pages that have been googled have to be kept alive and accessible--at some point I'll see if a 301 permanent redirect can be done on them to update Google's links without affecting Page Rank.

Expect to see a lot more in the coming weeks.