Brijj.com - Professional Networking

Naukri launched brijj.com, a networking site for professionals. With this, the company has entered the big league of web 2.0 companies, focusing on user contributions. The focus will be more on providing a platform for passive users to reach-out to other professionally linked people. I like the UI as i find it refreshing. The next step, i guess would be to increase stickiness on the site, thereby increasing the number of members. By concentrating on professional networking, naukri will have an edge over its competitors, as it has carefully not drifted away from its domain. As for the advertisers, they won't get a better place than naukri & brijj combined to reach out to the largest database of professionals in india. Way to go ....

Simplicity and usefulness is the essence of a good product

Few days back, a friend suggested an offline wiki to take notes. I decided to check it out and found it really cool. The best thing about it is its simplicity; yet it is very useful. That according to me is the essence of any good product.
Tiddlywiki is a very powerful concept. You can work offline in an html file and save it. It uses html, css and javascript and runs on any browser. Some important features that is provided by it are,

• Formatting and indenting
• Various interface options
• Backup and autosave
• Embedded images
• RSS feeds
• Keyboard shortcuts.
• Search Facility
  

To use it, all you have to do is save the source of Tiddlywiki and start using it. It saves the file when you save the note. All it does is allows you to re-write the file. Your notes are saved under headings provided by you. These can be browsed from the list provided at right side of the page. The best part is this small file allows you to search through the content.
If you have little html or javascript knowledge, you can play around with the code and make the most of this concept. I must admit ... some really good thinking here by Jeremy Ruston, who originally created this wiki.
The success of an idea, or for that matter, a product, depends on its usefulness and the ease/simplicity with which it can be put into shape. That's what matters.. other factors can follow.

My homepage

Had put up an index page on my website few days back. It is raw though... will work on it whenever i get time. Visit me @ sujithnair.com . Got to look for a decent hosting service with lamp support.

Web 2.0 in a restaurant ???

Ever wondered what will happen if some of the web 2.0 (or so called) concepts are used in a restaurant or in a dress store ? Well, i was just thinking about it when i came across these funny thoughts, yet interesting ones.
Lets take the very popular amazon's baby "wisdom of crowd" concept if used by KFC. I order for a small bucket of crispy chicken and they tell me that most of the consumers who prefer this have it with salsa sauce. We'll, i woudn't mind spending another extra 7 bucks for it. Or better still, they offer me a discount on chicken original receipe as the same group of people who prefer crispy chicken have a taste for the original reciepe as well. Now what will KFC gain out of it? They will increase sales on other items and at the same time build an impressive alternative list of items. What will i gain ? I'll shed the resistance to try new items and satisfy my taste buds.
Scene 2. I walk into levi's and try out some cargos. They offer me a discount on cool back-packs. Here, the probability of my buying this piece will be more as they are promoting an item after knowing my preference (cargos). But why back-packs ?? Because most of the customers who bought cargos show interest in back-packs as well. This has to be in their records.
Another popular baby of web 2.0 is "tags'. It is the result of user classification. If dress stores define sections based on user preferences, rather than product types, chances are high that the customer will end up buying more of different products. If a section has collection of all sporty products right from Ts to shoes to sweat bands, a customer who prefers sports apparels may end up picking each one of these. She may loose interest if she has to go to a separate shoes or sports accessories section. The point is to arrange or classify sections basis user interests and not product types.
Pardon me for my understanding and interpretations (if wrong) as I am clueless about the functioning of these industries. But having said that, i sometimes feel that it must be the other way round... that is, may be the online industry is taking a clue from other industry. You would agree that if you closely study the marketing and product strategies of these industries. Come to think of it, there would be endless opportunities if experiments are done basis learnings from other industries. The fact can be that these concepts were already prevalent in other industries but didn't have such flashy names or jargons.

JobSearch 2.0 Beta

It is over a week since we launched the job search beta on our site. Some interesting feedbacks have come our way. Job search 2.0 beta project was undertaken as a stealth project, and am happy that most of the projects undertaken as stealth are able to see the light of day. With support from all depts, especially product, we are able to showcase some good technical stuff. Hope to keep it rolling. Some more interesting stuff in pipeline. Watch out for this space :-)

Vector Space Model

Saturdays are dull in office. Not interested in doing the regular work, i decided to do little bit research. Searching for a specific algorithm, i stumbled upon the good old "vector based model" for search. Thought about brushing up my knowledge on the same and write a comprehensible writeup on it.
In this model the frequency of words is the catch. In the Salton's vector based model both local and global information is considered.

Term Weight = Wi = tFi * log (D/dFi)

Here,
* tFi = term frequency (term counts) or number of times a term i occurs in a document. This is the local (within a document) information.
* dFi = document frequency or number of documents containing term i
* D = number of documents in a database.
* i = the term
dFi/D = is the probability of finding term i in total documents D. Therefore, D/dFi is also called as the "inverse document frequency" or how many times the term i appears in the entire database.

Ok, now lets go through a simple example. Have referred the same from an example found in an online paper.

Lets say, there are 3 documents (D=3) with the given data,

D1 = "the Lotus is in the pond"
D2 = "Garden has a pond"
D3 = "Lotus is a flower in the center"
The query Q = "Lotus Garden Flower"

To start, we must do the following,

1. Take out all unique words from the three documents and sort them.
2. Determine how many times each word appears in the respective documents. For eg, here, "Garden" appears 1 time in D2, "Lotus" appears 1 time each in D1 and D3, "the" appears 2 times in D1 and 1 time in D3.
   Therefore,
   The term frequency tFi for "Lotus" in D1 is 1, D2 is 0 and D3 is 1.
   The document frequency dFi for "Lotus" is 2 (found in D1 and D3).
   Document frequency (D/dFi) for "Lotus" is 3/2=1.5
   Inverse document frequency (IDF=log(D/dFi)) for "Lotus" = 0.18
   IDF is a global value for each term.
3. Now that we have the parameters for the formula, we can calculate the weight for term 't' in each documents 'd'.
   Wt,d = tFi * IDF
   So (Wt,d) for "Lotus" for D1 = 1 * 0.1761 = 0.1761
   (Wt,d) for "Lotus" for D2 = 0 * 0.1761 = 0
   Similarly, (Wt,d) for all documents and the query is found.
   (Wt,d) for "Lotus" in query Q = tFi * IDF = 1 * 0.1761 = 0.1761
   Pls note, if the term frequency (tFi) is 0, then the (Wt,d) is also 0.
   Thus we have weights (Wt,d) for Q, D1, D2 and D3.
   
   
   
   
4. Now, we find out the vector lengths.
   
   
   
   
5. Now, we calculate all dotproducts except all zero products
   
   
   
   
   
6. Now, we calculate the similarity values
   
   
   
   
7. In the end, we will sort on the cosine values in descending order and the document in the top will be the highest ranked. In this case, the rank will be
   
   Rank1 : D2 = 0.39
   Rank2 : D3 = 0.32
   Rank3 : D1 = 0.15
   

The above explained example is a simple example just to demonstrate the vector space model. In a real search engine, there are other factors which needs to be considered. The important ones are -

• Stop words - While extracting all unique terms from the documents, the prepositions are removed. Here for eg - a, has, in, is, the, where. Also special characters like ',' '.' '-' etc can be handled.
  
• Handle different forms of the word. As here a pure keyword search is done, trimming the term down to its root word and then matching it to the query will be a good idea. A good stemming algorithm will help. Stemming will help in matching all words like develop, developer, developers etc.
• It is also important to know where within a document a term is found.

The vector space concept is widely used in search engines. MySQL is one good example which is inspired of this model and has implemented it in its fulltext engine.

In the MySQL implementation an application uses database tables consisting of N rows. Each row corresponds to a document, where

1. L_{i, j} = (log(dtf)+1)/sumdtf; i.e., local information based on logarithmic term counts
2. G_i = log((N-nf)/nf); i.e., global information based on probabilistic IDF
3. N_j = U/(1+0.0115*U); i.e., normalization based on pivoting

Here,

dtf = number of times the term appears in the document (row)
sumdtf = sum of (log(dtf)+1)'s for all terms in the same document (row)
U = number of unique terms in the document (row)
N = total number of document rows
nf = number of documents (rows) containing the term

The following equation is thus derived,
w = (log(dtf)+1)/sumdtf * U/(1+0.0115*U) * log((N-nf)/nf)

Well, now that we have understanding of this model, we can always tweak it as per our use and derive a new algorithm. Gonna be fun :-)

Concept ... i m working on.

"Every man belongs to some community ! Similary, Every word should belong to one or more group or category"

"More the strength of communities, more stronger will be the chance of a man fitting into a community. Same is true for a word also....In other words, more the data, more chance for a word to fit into a pattern. "

Stealth Projects

Its a fresh day, there is a bunch of Great Ideas, you have high energy level, it is time to do some action... before it even starts, your ideas are exposed to a number of double barrel guns shooting continuously with reasons why we should not work on the "Great Ideas". Now what ???
One option can be to lay back and work only on pre-approved ideas or "requirements" in other words. But hey, we are innovators and we believe in our ideas. So we decide to take the other path and start work on them. Well... in STEALTH mode though for obvious reasons :-)
This is a new approach i had started around 4-5 months back which is yielding rich dividends. Had started this in the NI team and slowly implemented in the search team as well. I can see my boss endorsing this approach and quite happy with the developments.
Though it has been a success so far, i am expecting a liitle more. I wish one day we'll have more self motivated engineers who will start their own research project with a bunch of other developers, and deliver stuff which our competitors can't even think about. That day, there won't be anything like 'stealth'...
Right now, I am soon going to start a new big stealth project which will be a great research work. Needless to say, i am quite excited about it. Will write about it once it is over. Well, that's why it is in stealth mode ;-)

Search by "Intent"... not "Content"

I had been busy with some developments in my search product, but nothing interesting enough. My team has just started putting into shape one of our long pending objectives. Hope to release it by end of feb.
Now, after seeing some action happening towards this direction, am off to visualize and analyze my next aim. That is "Search by intent" . I have always felt that internet search which looks good today is not even 10% of what it can be. There is a lot of intelligence that can be introduced. I am referring to behavioural intelligence here. Now what's that ? Hmm.. it is my term ;-) Well, it is the intelligence derived from user behaviour. Nice and patient analysis of your search dump will help you develop this intelligence better.
Coming back to search... if we can know the intention of the user, we can provide much better results. If we crack this problem and are able to covert CONTENT into INTENT, we would have solved half the problem. Many of the big search engines have started researching in this line.
Yahoo has already launched the beta version of their search engine "Yahoo ! Mindset". It provides a cool slider to dynamically rank results. It shows results basis its informational or commercial aspect.
I happened to stumble upon a blog which writes about Matt cutts (google) hinting that google is doing scientific research in this field. I won't take it as a joke. We can soon see some intent oriented search happening in google.
Users intention can be known and taken care off by,

• Search Dump - A proper analysis of search dump will throw light on the "intent" behind every search. This study has to be w.r.t the results clicked upon by the user from the search result set.
• Personalization - Proper tracking of user activities can open a whole new world of knowledge for search engines. If I visit a search engine every week and click on a set of links, the search engine should damn well know what i am looking for every time. And if you have the user profile with you, then what else can you ask for.
  
• NLPs (Natural Language Processing) - Natural language search is what every user is comfortable doing. The engine should be able to derive the intent of the user from the text provided in the search box.
• Phrase Searches - The emphasis should be more on phrase searches. The more the user writes in the search box, more clear is his/her intent. Ofcourse, the search engine should be geared up to handle the shrink in result set due to large phrase search. Here, the result boosting algorithms come into play.
• Wisdom of crowd - Now, this one is a much talked about concept out of the web 2.0 books. Entire trail of all user activities should be logged. We can know a completely new user's intent by comparing his/her search with searches made by other users, and establishing a pattern. If most of the users search for "apache" and click on pages dedicated for apache web server, we can assume that the new user is also interested in apache web server and not in apache tribes. Here, the results pointing to the former can be rated higher in the search. Ofcourse you have to include all results though.
  

I had read somewhere about a quote from Peter Norvig, director of research for Google talking about how Google returns results to search query.
"We want to do a better job of understanding the user's intent and the content provider's intentions,"
and
"We mostly rely on matching keywords, but we'd like to get closer to matching the intent."
The same sentiments are echoed by Adam Sohn from microsoft,
"If someone is searching for 'Jaguar, the smarts to distinguish between 'he's looking for a car' and 'a big cat in the jungle' - that's coming."
Here, the "intent" of the big players in the search business is clear.

A major chunk of work in a search engine development goes into large set of data collection and analysis. Better log analyzers and information system should be in place. A thorough research on this information set is carried and the outputs serve as inputs for algorithms. A continuous analysis of user behaviour is important to monitor pattern deviations. Studies have revealed that phrase searches have become more common as compared to searches done 6-8 years ago. No doubt therefore that n-grams are hot again.

I have got to do a lot now. I am yet to have a good information system. There is lot of tracking which still needs to be done. More i think of it, more i realize that the entire data management, right from tracking is a different area. I, as a techie won't be able to do much justice. I'll be more interested in the output of these studies to convert them into complex algorithms. Hmm... till we have a separate team for these studies, tech has to manage it. Can't complain though... it is giving me much needed exposure.

A fresh day for naukrigulf.com

It has been a usual day with some small achievements giving you immense joy. One of these was making a customizable homepage for naukrigulf.com job seekers. It was my dream for last 2 years to develop a highly customizable section for job seekers. With some web 2.0 knowledge sinking in, i was able to visualize it in entirety. And i must say, i have plans for the next one year already well thought through. But, the smart way to go about it will be step by step. I know people will argue that such features and facilities will be overwhelming for a job seeker, but the fact is, you do not loose anything by providing an additional feature to them as long as their normal course of action on the site is not affected. I think, we should give them more and more so that they know that we care, and are continuously working to make things better for them. A user should feel invited to come to your site.
Have always felt and have said a hundred times that the future of web lies in customization, personalization and collaboration. It's better to experiment on this right now rather than take desperate steps when the whole world has done it. If you have the technology and vision, who is stopping you ? I like google for this approach. They keep producing stuff which many of us may not find useful. However, there will be some others who will be really thankful to google for coming up with such products.
Idea is not to satisfy the top 10 or 10% of your users 100%.. But i think, the idea should be to satisfy them 90% and satisfy the remaining 90% atleast 10%. You cannot afford to ignore the long tail.
My second step will be to provide as many components as possible to the job seekers. I'll definitely propose to make the homepage compatible with third party components as well... ofcourse, those components should make sense on my site.
I'll concentrate more on such components which will help job seekers to find and apply to jobs seemlessly.
My first step, i would like to believe, was in the right direction. It would not have been possible without the help i got from my team mates and my collegues in other departments. It could face the light of the day because my top management and collegues also shared my belief and vision. there are lot many steps to be taken further. It will be a long walk... give me one more month, the next version will be another big step in the "Right Direction".

Component Based Architecture & Service Oriented Architecture

The future of web is in sharing. It is time, web sites providing online services should gear themselves for this. It is better than to slog inorder to avoid competition. In coming years (or months.. who knows), online features will be rated on their plugability/mashability, along with useability and speed.
Component Based Architecture allows you to do exactly that. The software architecture should be such that each feature/product is defined as a "component". These components can then be used for different services. "SOA" is thus born... taking you to the world where web applications can be shared.
A web product is so designed and developed, that it can be used anywhere seemlessly. Few points that needs to be kept in mind are,

1. A component should be independent
2. It should be pluggable
3. It should be reusable
4. It should be mashable
5. It should be asynchronous
6. It should be secure

I stress upon the term "asynchronous" as i personally prefer it :-)
The best way to expose your services is through SOAP and REST. When designing your application software, do it keeping the above two in mind.
The services offered through these components should be such that these are available at the client, as well as at the server level. To put it simply, it should allow a presentation layer (for eg, javascript) integration, as well as middle layer integration.
When developing services for the presentation layer, always write APIs. If you need, you may go a step ahead and write a client for it as well. While doing that, get a peek into "rich internet applications" (RIA) as well.
To start, you may use javascript/iframes for a presentation layer integration.
I see this as the future. Your web application components will not be confined to your websites or your products. It will go beyond that and help make better products and services. Everything tomorrow will be personalized and customizable. Design your applications in such a way that you are geared up for this demand. You do it today, and you are already ahead of your competetion. A user should be able to decide what he/she wish to use and from where... that means, "the web would become programmable". Enterprise soultions with OpenSource is the mantra.

Few books worth a read,

• Service-Oriented Architecture (SOA): Concepts, Technology, and Design
  by Thomas Erl
• Service-Oriented Architecture : A Field Guide to Integrating XML and Web Services
  by Thomas Erl
  
  

My allepy visit... houseboat in the backwaters of kerala.

Search Engine - First Steps
With the web developer community already looking into next generation online solutions, it becomes imperative that user experience is seemless and tools are provided for the user which ensures optimum value with respect to time spent online. Search engines within applications have thus become integral part of every system. Today, i'll be discussing about the important factors to be kept in mind while developing a search engine for your database. Most of this i have learnt while working with my current organization. There are also stuff which i do not agree with, which i think will only result in loss of user base in the long run.
Here are some points which i think (and ofcourse experienced) will help you start.
Identify the user base - This is important so that you can provide further guidance to the user from the search results page, basis the search conducted. The search engine should be like a restaurant waiter :-) It should ask you everything about your taste (read requirements) and then offer you exactly what you had asked for. If the food (read results) does not taste good, you are unlikely to come back.
You should also provide guidance and suggestions so that the user gets to see the best result within no time. This set of facility can be divided into two.
A. When no or less number of results are returned -

1. Users should be guided to broaden the search if their search criteria is too specific by using clouds.
2. You can also show related results by doing "content mapping" in the background.
3. Check for spellings and suggest correct words.
4. Other concepts like "stemming" can be incorporated in the algo to show more results as per the requirement.
5. Suggestions - Show suggestions basis your domain intelligence. This should be purely basis the historical data/logs of search you have.
   

B. When too many results are returned -

1. Clustering - Categorize or classify the result set. This will help users to dig into more relevant results as per their requirement.
2. Predefined Categories - Show a list of predefined categories.
3. Search within search - Allow users to conduct search within search, so that they can drill into more relevant and specific results.
4. Response time - Fast response to a search is more of a necessity these days. User should not be left waiting for the results. Important sections of the search results should load within 1-2 seconds is my recommendation. Care should be taken that by providing the above listed facilities, the response time is not affected. Search engine should be divided into various segments, and the best technology should be used for each one of these. A s/w developer may want to use different languages for different segments of the engine considering the processing time, easy of change etc.
   

Other factors important for a search engine are,
Page layout - The UI team should ensure that the page layout is such that the organic results should not be contaminated with advts and other paid results. Premium listings, advertisements etc should be separately shown. Also the construction should be such that the download of page is controlled basis importance of each section on the page.
All actionables on the page should be prominent. It should be analysed if these actions can be allowed on the same result page through DHTML, CSS etc. AJAX can be of great help here.
Navigation - Research has found that most of the users do not go beyond page 1-2. Therefore it is imperative that these pages should show the most relevant/fresh results as per the requirement. Also, prefetching of pages will result faster navigation across pages.
Domain Intelligence - Search engine developers and product managers should continuously monitor search logs and derive important information on user behaviour out of them. Most of the features related to search can be developed by studying the search dump. I would recommend that product managers should not get influenced by popular features on other search applications. They should rather study their own users' behaviour through search dumps and then conceptualize new features, sections etc.
S/w developers should be able to fine tune their algo by looking at search dumps and search performance logs.
Logging - Enhancements/ tuning of search engines is continuous. Therefore it becomes all the more important that all aspects of the search should be logged so that post enhancement analysis can be conducted and corresponding actions can be taken.
Research - Its an important part of any search engine development. Developers should be aware of latest technologies and findings. Right now web 2.0 is hot. It should be analysed with respect to your user base and the best suited technology/concept should be adopted. Choice of database and language should be carefully done.
Besides the above listed points there are other concepts like tagging, mashups etc which may contribute to your search peripherals. Now mark my words here when i say "peripherals". A search results page should not get contaminated with overwhelming hi-tech concepts. It should be simple and seemless.
I keep surfing net for niche search related technologies/concepts. Couple of websites which i frequently visit for search related news are searchenginewatch.com and battellemedia.com. The later one is John Battle's blog. His book "The Search" is worth a read. He keeps a close watch on developments in search engines, especially google. I also keep visiting jeremy's blog as it has some interesting stuff on MySQL.

Search Engine - Supervised Rankings

Was going through various online journals on search engines when i stumbled upon this document published by mondosoft. Its interesting as they have argued the importance of human intervention in search engines. I wish i could attach the entire pdf here, but you may be able to download it from here
This document mainly covers points about gathering user behaviour data, interpreting and analyzing log data, providing informative results etc. Its worth a read. I am trying to get more whitepapers from them.
We have also concluded on similar findings. Our bottleneck lies in implementation. Don't even think that we are technically incapable. Its just that sometimes too much discussions amongst bright people results in a very stringent priority list. The best approach could be to have a clear plan for 6 months and work towards it. Its however easily said than done. For us, market feedback and faster turnaround time in crucial. We are always on our toes.
I am planning to write a small journal on search engine implementation soon....

php extension

Had always thought of exploring the php extension creation part. Although i have always encouraged my team to go for it wherever it makes sense, never got time to sit down and write my own first piece of extension. Today i sat and decided to write my first php extension. Doing this on a saturday definitely helped as i didn't get distracted with daily fire fightings and meetings.
The whole beauty of writing the extension and making it part of the php you have just installed from source is mesmerizing. I was able to easily write an extension and run it through the command prompt. Running it from the cli was seemless, however it didn't work from the web server. Got to check it out now. Have to rush back home. Will lick this problem later. For now, the boost that i got from writing a small extension is enough to motivate me to write a useful extension soon and contribute to the community. Well here is the Zend url which helped me with my first shot at extensions.

labs.naukri.com
Have started conceptualizing naukri.com developer-network under labs.naukri.com umbrella. Have listed out various sections and resource links for naukri labs. Am planning to start it, fill it with content and then do an SEO for the same. Today, i sat with sonali to discuss its layout. Hopefully she will start working on it from monday onwards. Lets see how it goes. I plan to have resource links for PHP, MySQL, Apache, AJAX, Advanced CSS, Lucene etc on it. Also, have planned an interesting section called "wine tasters" wherein the regular chosen ones will be allowed to taste....Oops..., test and try out products/features in making. I have always believed that true contributions happen through the users. I am excited about it. Not much work, but opportunity to visualize something like this and put it into shape is really motivating.

Finished my days work. Just going through different websites using latest technology. Visted netvibes.com and found it interesting. But i sincerely wonder, how useful these "myspace" kind of concepts would be. Whole lot of it depends on how the entire web progresses crossing limitations of bandwidth. I'll definitely love to create my workarea on the web and stick to it provided i have limitless bandwidth :-)
Web can be so more useful if it continuously reaches out to different medium of communications. Its happening and a lot would depend on how we take it forward. Hmmm... long way to go. Gonna have some coffee..

Web 2.0

I don't know if i should use this (web 2.0) term, but the hype associated with it has definitely given a wider perspective to the web and its components. The emergence of AJAX is the biggest achievement of this era. Although many of us had used javascript to fetch server side data as as early as 6 yrs ago, somebody had to come out with a fancy name like "AJAX" to make it popular. Same is true with "Web 2.0"
The most important achievement of this era was the shift in the mindset of web service providers from proprietory to service oriented approach. Emergence of Opensource has nodoubt acted as a catalyst in this possitive shift. This is now giving way to Service Oriented Architecture (SOS) and Software As A Service (SaaS) theory.
I, as a techie find the emergence of collaborative softwares as a strong point. We have to go a long way. APIs in the web community was just a beginning. I strongly advocate that software architecture should be designed in such a manner that every important feature is available with the help of APIs as a plug-in module. Mashups ofcourse tells you how to make most use of it.
Another important approach is to reduce the number of steps (interfaces) from a web users first click till his/her last action. An intelligence use of JavaScrit, DHTML, CSS and AJAX will help one achieve this goal. The user should be allowed to take all actions on the single page itself, rather than navigating to different pages. A Desktop like feel will soon be in demand. Rich user experience along with fast downloads will become a necessity soon. Ok... i'll stop acting like nostradamus :-)
Going to do some work now. Time is less and i have got to implement so many things. I am planning to use it as a canvas for all my web 2.0 experiments. I am sure, its gonna work.