获取HTTP请求然后处理HTML响应的最佳方式

我正在处理一个项目，该项目需要从其他网站导入数据，然后处理此响应，以便能够将其存储在数据库中，并将其与其他结果进行比较。

这些网站中的大多数都不提供json或xml提要，所以我发现自己被迫获取HTML文档。

首先，我开始用CURL获取它们，并用PHP DomDocument&Xpath。但正如我所注意到的，这个过程可能非常缓慢。因此，我开始寻找另一种可能更快、更可靠的解决方案，即使它使用的是另一种语言，如：python、ruby、perl、C等

代码示例：

这只是一个测试代码。在开始项目之前。

初始CURL:

public function curlInit($connId, $url , $postString){
    $this->multiConn[$connId] = curl_init($url);
    curl_setopt($this->multiConn[$connId], CURLOPT_RETURNTRANSFER, true);
    curl_setopt($this->multiConn[$connId], CURLOPT_HEADER, false);
    curl_setopt($this->multiConn[$connId], CURLOPT_ENCODING, 'gzip,deflate');
    curl_setopt($this->multiConn[$connId],CURLOPT_POSTFIELDS, $postString);
}

执行请求：

public function multiExec(){
    $this->multiAddHandler();
    $running = null;
    do {
        curl_multi_exec($this->multiHanler, $running);
    } while ($running);
    $i = 0;
    foreach($this->multiConn as $key => $v){
        $this->multiRespose[$key] = curl_multi_getcontent($v);
    }
}

这是执行请求的代码：

public function parse(){
            // first fetch & process the categories
    $this->getCategoryElementList("li");
            // then fetch all events on each category.
    $this->curl->multiExec();
            // parse the events response
    $this->processEvents();
            // after that i have to parse the details of each event.
}

现在，解析HTML响应：

public function getCategoryElementList($tag){
    foreach($this->categoryIdsArr as $group){
        $domElement = $this->getElementById($group);
        $catList = $domElement->childNodes;
        $this->categoryElementList[] = $catList;
        foreach($catList as $cat){
            // temp Var , to check if the subcat_id is autogenerated, so don't init a curl connection for it
            $autoGenSubCatIds = array();
            if($cat->nodeName == 'li'){
                // -- Getting the category name -- //
                $catNameSapn = $this->searchElement($cat, "span", "class", "nav-special-name");
                if(empty($catNameSapn->item(0)->nodeValue)){
                    $catNameSapn = $this->searchElement($cat, "span", "class", "nav-region-name");
                }
                if(isset($catNameSapn->item(0)->nodeValue)){
                    $_categoryName = $catNameSapn->item(0)->nodeValue;
                }
                // autogenerate subcat_id if not exists 
                if($catNameSapn->item(0)->childNodes->item(0)->nodeName == 'a'){
                    $aTag = $catNameSapn->item(0)->childNodes->item(0)->getAttribute("href");
                    $aTag = split("/",$aTag);
                    $_categoryId = $aTag[4];
                }elseif($catNameSapn->item(0)->childNodes->item(0)->nodeName == '#text'){
                    $tempId = 0;
                    array_walk(str_split($_categoryName), function($value, $index) use (&$tempId){
                        $tempId += ord($value);
                    });
                    $_categoryId = ($tempId);
                    $autoGenSubCatIds[] = $tempId;
                }
                // -- End getting the category name -- //
                $this->arrRes['category'][$_categoryId]['category_name'] = $_categoryName;
                $this->arrRes['category'][$_categoryId]['category_id'] = $_categoryId;
                $subCats = $cat->getElementsByTagName("a");
                foreach($subCats as $subCat){
                    $_subCategroyName = $subCat->nodeValue;
                    $aTag = $subCat->getAttribute("href");
                    $aTag = split("/",$aTag);
                    $_subCategoryId = $aTag[4];
                    $this->arrRes['category'][$_categoryId]['subcat'][$_subCategoryId]['subcat_name'] = $_subCategroyName;
                    $this->arrRes['category'][$_categoryId]['subcat'][$_subCategoryId]['subcat_id'] = $_subCategoryId;
                    if(!in_array($_subCategoryId, $autoGenSubCatIds))
                        $this->curl->curlInit($_subCategoryId, "https://********************.com", "Ids=$_subCategoryId&stId=4&page=0");
                }
            }
        }
    }
}

我的问题不仅仅是关于这个代码。我正在寻找最好的方法来计算连接并解析它们