Built a web crawler , C/C++ Programming

To develop a web crawler such that when given a base URL, it will traverse the entire web tree and then build an index of keywords and what URL link they appear on.   The web crawler is a server-side program where it can begin called the seed.  As the crawler visits the URL, it identifies all the hyperlinks in the page (eg: detecting the " < a href ..." links) and adds them to the list of URLs to visit, called the crawl frontier.   URLs from the frontier are then visited by your server-side program to look for a few keywords (eg: raffles, award, alumni etc) initially on this website and stored their corresponding URL links where these keywords can be found in a file.  

Below is a sample of the data stored in the Server for subsequent quick search.

Another interactive client and server socket programs using C++ can be developed to allow clients to query with the search keywords of the website and obtain a server's reply of a list of URLs where the keyword is found.  The server program developed must be able to handle multiple queries and hence be able to search through its data file for a correct response.  The server program should also allow continuous enquires until the customer enters quit.  

The communication between the client and server machine can be any bidirectional interactive protocol.  An example is to use socket programming where network endpoints (IP address and port number) are represented as sockets.

 

When creating the server application, you must follow these steps:

  • Create a new socket by typing: socket().
  • Bind an address (IP address and port number) to the socket by typing: bind. This step identifies the server so that the client knows where to go.
  • Listen for new connection requests on the socket by typing: listen().
  • Accept new connections by typing: accept().

Often, the servicing of a request on behalf of a client may take a considerable length of time. It would be more efficient in such a case to accept and deal with new connections while a request is being processed. The most common way of doing this is for the server to fork a new copy of it after accepting the new connection.

The "responsing" Server listens on a port and waits for client's request. Based on the Client's questions, the Server will response appropriately by looking up a data file stored at the server's end.  

Posted Date: 2/22/2013 6:38:02 AM | Location : United States







Related Discussions:- Built a web crawler , Assignment Help, Ask Question on Built a web crawler , Get Answer, Expert's Help, Built a web crawler Discussions

Write discussion on Built a web crawler
Your posts are moderated
Related Questions
A: No.       While realloc() need to copy the allocation, this uses a bitwise copy operation, that will tear several C++ objects to shreds. C++ objects must be allowed to copy t

Yes. "Memory pools" are useful in many situations. The bad news is that I'll need to drag you through the mire of how it acts before we talk about all the uses. Firstly, recall

Write a C program to input your full forename and full surname. e.g James McCarren and display in one string your complete initial and surname i.e J McCarren . Your initial should

At a shop of marbles, packs of marbles are prepared. Packets are named A, B, C, D, E …….. All packets are kept in a VERTICAL SHELF in random order. Any numbers of packets with thes

Operator Overloading Fundamentals The C language uses the concept of Operator Overloading discreetly. The asterisk (*) is used as multiplication operator as well as indirect

Write a main program that uses these system calls to echo the typed characters. The pseudo code will look something like: void traphandler ()  {    if (R0 == 0) { // read sys

The project is to design and write a C++11/FLTKgame program with a graphical user interface.The game is based on "pancake sorting," as described at http://en.wikipedia.org/wiki/P

Note: Please refer to the Subject Outline for details regarding the assessment of the advanced assignment. The Problem You are to investigate the use of data structures an

Explain the Macros? Preprocessor' is a translation stage that is applied to your source code before the compiler proper gets its hands on it. Usually the preprocessor performs

Aim: To implement program to calculate area of cube using inline function. Code: inline void area_cube(float side) {             float area;             are