如何抓取HTML页面来创建一个总的主观站点得分

我建议使用python的urllib。

抓取网页

使用Python很容易通过HTTP获取标准Web页面:

进口urllibF = urllib.urlopen("http://www.python.org")
S = f.read()
f.close ()

——这是从这里

然后使用python的html解析器

在指定的站点(sitename)中查找网站上所有html页面中的各种字符串(a, b, c, d, e)和特定的命名javascript文件(javascriptfile.js)

在python中，您需要使用urllib。这将允许您轻松地与Http服务器通信。然后您将需要查看regexp，这将允许您执行爬行和字符串搜索。由于大多数服务器没有开放索引，您需要找到<a>标记，然后去掉它们指向的所有内容，然后抓取一个新的目的地。

从锚标记中获取Href属性

比较域，确保它们是相同的或一个相对路径(以'/'开头)

重复过程

你可以看看'beautifulsoup'来帮助你。它将为您完成所有阅读HTML的艰苦工作。Beautiful Soup

甚至可以帮助搜索你的字符串。

如果在单个页面上没有找到javascript文件，则将该页的名称/url输出到文件中，然后继续爬行

您可以再次使用Beautiful Soup或RegEx查看他们是否实际上将其包含在<script src='urltofile'>页面上。然后把你正在抓取的当前页面写到一个文件中。

根据每个字符串在页面上出现的次数创建一个总分(每个字符串1分，每个字符串2分)，等等

这将完成所有你正在爬行的页面，使用Regex你可以计算多少次文本模式的特定实例出现，所以你只需要把这些添加到字典和得到你的结果。也许可以创建一个映射，所以score = {'a': 10}; IF a FOUND: points += score['a']*occurences。

很好的Regexp参考:Regexp Info

第一点是这样的(在PHP中):

• 加载html页面-您可以使用file_get_contents()或curl(推荐)用于此
• 做一些preg_match 'es在网站上寻找事情a,b,c和js脚本名称或使用使用http://www.php.net/manual/en/book.dom.php加载页面作为XML和做一些xpath的对他们(http://www.php.net/manual/en/book.dom.php#93637)(推荐)

只有这样你才能转到第2点和第3点

我不太明白你的问题，但我想这会有所帮助:

创建一个简单的爬虫，它将数据插入到数据库中。然后在另一个PHP文件中从表中选择这些行，然后找到爬行文本的特定部分，然后为它们指定所需的值。然后更新数据库

下面是一个PHP的爬虫代码:

<?php
$urls = array("http://www.chilledlime.com");
$parsed = array();
$sitesvisited = 0;
mysql_connect("localhost", "username", "password");
mysql_select_db("db_name");
mysql_query("DROP TABLE search;");
mysql_query("CREATE TABLE search (URL CHAR(255), Contents TEXT);");
mysql_query("ALTER TABLE search ADD FULLTEXT(Contents);");
function parse_site() {
    GLOBAL $urls, $parsed, $sitesvisited;
    $newsite = array_shift($urls);
    echo "'n Now parsing $newsite...'n";
    // the @ is because not all URLs are valid, and we don't want
    // lots of errors being printed out
    $ourtext = @file_get_contents($newsite);
    if (!$ourtext) return;
    $newsite = addslashes($newsite);
    $ourtext = addslashes($ourtext);
    mysql_query("INSERT INTO simplesearch VALUES ('$newsite', '$ourtext');");
    // this site has been successfully indexed; increment the counter
    ++$sitesvisited;
    // this extracts all hyperlinks in the document
    preg_match_all("/http:'/'/[A-Z0-9_'-'.'/'?'#'='&]*/i", $ourtext, $matches);
    if (count($matches)) {
        $matches = $matches[0];
        $nummatches = count($matches);
        echo "Got $nummatches from $newsite'n";
        foreach($matches as $match) {
            // we want to ignore all these strings
            if (stripos($match, ".exe") !== false) continue;

            // yes, these next two are very vague, but they do cut out
            // the vast majority of advertising links.  Like I said,
            // this indexer is far from perfect!
            if (stripos($match, "ads.") !== false) continue;
            if (stripos($match, "ad.") !== false) continue;
            if (stripos($match, "doubleclick") !== false) continue;
            // this URL looks safe
            if (!in_array($match, $parsed)) { // we haven't already parsed this URL...
                if (!in_array($match, $urls)) { // we don't already plan to parse this URL...
                    array_push($urls, $match);
                    echo "Adding $match...'n";
                }
            }
        }
    } else {
        echo "Got no matches from $newsite'n";
    }
    // add this site to the list we've visited already
    $parsed[] = $newsite;
}
while ($sitesvisited < 50 && count($urls) != 0) {
    parse_site();
    // this stops us from overloading web servers
    sleep(5);
}
?> 

祝你好运!

只是另一个选项html5lib一年多以前，它似乎是解析HTML的好选择。参见:http://code.google.com/p/html5lib/wiki/UserDocumentation

开始吧，处理这个搜索结果页面的示例:http://index.hu/24ora?tol=2010-08-25&ig=2011-08-25(这是匈牙利语)它将提取搜索结果的数量

from datetime import datetime, timedelta
from html5lib import treebuilders, treewalkers, serializer
import html5lib
import re
import urllib2
import sys
def openURL (url):
    """
    utlitity function, returns (page, url)
    sets user_agent and resolves possible redirection
    returned url may be different than initial url in the case of a redirect
    """    
    request = urllib2.Request(url)
    user_agent = "Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5"
    request.add_header("User-Agent", user_agent)
    pagefile=urllib2.urlopen(request)
    realurl = pagefile.geturl()
    return (pagefile, realurl)
def daterange(start, stop, step=timedelta(days=1), inclusive=True):
    """
    utility function, returns list of dates within the specified range
    """
    # inclusive=False to behave like range by default
    if step.days > 0:
        while start < stop:
            yield start
            start = start + step
            # not +=! don't modify object passed in if it's mutable
            # since this function is not restricted to
            # only types from datetime module
    elif step.days < 0:
        while start > stop:
            yield start
            start = start + step
    if inclusive and start == stop:
        yield start
def processURLindex(url):
    """
    process an url of an index.hu search result page
    returns number of search results
    e.g. http://index.hu/24ora/?s=LMP&tol=2010-04-02&ig=2010-04-02    
    """
    (f, new_url) = openURL(url)
    parser = html5lib.HTMLParser(tree=html5lib.treebuilders.getTreeBuilder("dom"))
    tree = parser.parse(f)
    tree.normalize()
    for span in tree.getElementsByTagName("span"):            
        if span.hasAttribute("class") and (span.getAttribute("class") =="talalat"):
            return re.findall(r''d+', span.firstChild.data)[0]

def daterange2URLindex(term, start_date, end_date):
    urlpattern = "http://index.hu/24ora/?s=$TERM$&tol=2010-04-02&ig=2010-04-02"
    cum = 0
    for single_date in daterange(start_date, end_date):
        datestr = single_date.strftime("%Y-%m-%d")
        url = re.sub(r"'d'd'd'd-'d'd-'d'd", datestr, urlpattern)
        url = url.replace("$TERM$", term);
        num = int(processURLindex(url))
        cum = cum + num
        print "'t".join([str(num), str(cum), datestr, url])  

if __name__ == '__main__':
    if len(sys.argv) == 4:
        start_date = datetime.strptime(sys.argv[2], '%Y-%m-%d')
        end_date = datetime.strptime(sys.argv[3], '%Y-%m-%d')
        daterange2URLindex(sys.argv[1], start_date, end_date)
    else:
        print 'search index.hu within a date range; usage:'
        print 'index.hu.py [search term] [from date] [to date] > results.txt'
        print 'the date format is yyyy-mm-dd'
        print 'the output format is TAB delimited and will be the following:'
        print '[count of search results]TAB[count cumulated]TAB[date]TAB[search URL for that date]'
        sys.exit(-1)