[an error occurred while processing this directive]

<P>
<TABLE BORDER="0" WIDTH="100%">
	<TR>
		<TD WIDTH="21%"><IMG SRC="dllogo-smaller-xparent.gif" WIDTH="243" HEIGHT="112" ALIGN="BOTTOM" BORDER="0"></TD>
		<TD WIDTH="79%">
			<H1>The Stanford WebBase Project
		</TD>
	</TR>
</TABLE>
</P>
<P>Our DLI2 WebBase project builds on our previous Google activity. It has the following goals:</P>

<OL>
	<LI>Provide a storage infrastructure for Web-like content
	<LI>Store a sizeable portion of the Web
	<LI>Enable researchers to easily build indexes of page features across large sets of pages
	<LI>Distribute Webbase content via multicast channels
</OL>

<P>Some of the challenges we are addressing include:</P>

<UL>
	<LI>The huge size of the information space
	<UL>
		<LI>Very large URL space
		<LI>Content size
	</UL>
	<LI>Limited resources
	<UL>
		<LI>Disk space
		<LI>Memory
		<LI>Time
		<LI>Bandwidth
		<LI>Web server tolerance
	</UL>
	<LI>Continuous update and maintenance
	<UL>
		<LI>Smart crawling to maximize freshness and likely relevance
		<LI>Crawler parallelism
		<LI>Crawling robustness; interruptible crawls
	</UL>
	<LI>Making it easy for researchers to use the facilities
	<UL>
		<LI>Standard access API
		<LI>API for building new indexes
		<LI>Convenient distribution of entire content
	</UL>
</UL>

<P>The project is developing the following facilities:</P>
<P ALIGN="CENTER"><IMG SRC="webArchitecture.gif" WIDTH="640" HEIGHT="480" ALIGN="BOTTOM" BORDER="0">&nbsp;</P>

<OL>
	<LI>Smart crawling technology. This will allow us to crawl 'valuable' sites more frequently or more deeply. The
	measure of 'value' is, of course, itself an important research topic. Google's page rank is one such measure.
	<LI CLASS="Numbered">Web repository. This infrastructure will hold large numbers of Web pages, and will allow experimental
	search and analysis over that information. We are working on the following components:
	<UL>
		<LI CLASS="Bulleted">An archive to hold the (compressed) pages that our crawler acquires. This archive will enable
		random access via an index. The archive is also capable of streaming its entire content to a client.
		<LI CLASS="Bulleted">An index factory. This facility will enable researchers to easily add novel indexes to the
		Webbase. To use the index factory, the researcher implements a feature extraction engine. When presented with a
		Web page, such an engine computes some page property of interest to the researcher. Examples: reading level, genre,
		link property analyses... Once the engine is constructed, the researcher requests that the archive be 'streamed
		past' the extraction engine. Based on the engine's output, the index factory will add an appropriate index to the
		Webbase. This index will then be available to the community. Webbase will also keep the index up to date as the
		archive is refreshed through new crawls.
	</UL>
	<LI CLASS="Numbered">Wide-Area Web Data Distribution. Our distribution machinery will allow researchers everywhere
	to take advantage of our collected data. Rather than forcing all index creation and data analysis to run at the
	site where the data is located, our wide-area data distribution facility will multicast the archive's content through
	multicast channels. Channels may vary in bandwidth and content. Subscribers specify the parameters of their channel.
</OL>

<P></P>

<P>See Junghoo Cho's <A HREF="WebBaseFigs/index.htm">Powerpoint presentation</A> for more detail.

[an error occurred while processing this directive]