从php中的csv文件读取大数据

// for set memory limit & execution time
ini_set('memory_limit', '512M');
ini_set('max_execution_time', '180');
//function to read csv file
function readCSV($csvFile)
{
    $file_handle = fopen($csvFile, 'r');
    while (!feof($file_handle) ) {
       set_time_limit(60); // you can enable this if you have lot of data
       $line_of_text[] = fgetcsv($file_handle, 1024);
   }
   fclose($file_handle);
   return $line_of_text;
 }
// Set path to CSV file
$csvFile = 'my_records.csv';
$csv = readCSV($csvFile);
for($i=1;$i<count($csv);$i++)
{
   $user_email= $csv[$i][1];
   $qry = "SELECT u.user_id, u.user_email_id FROM tbl_user as u WHERE u.user_email_id = '".$user_email."'";
   $result = @mysql_query($qry) or die("Couldn't execute query:".mysql_error().''.mysql_errno());
   $rec = @mysql_fetch_row($result);
   if($rec)
   {
      echo "Record exist";
   }
   else
   {
      echo "Record not exist"; 
   }
}

处理大文件的一个优秀方法位于：https://stackoverflow.com/a/5249971/797620

此方法用于http://www.cuddlycactus.com/knownpasswords/（页面已被删除）在短短几毫秒内搜索1.7亿多个密码。

在苦苦挣扎之后，我终于找到了一个好的解决方案，也许它也能帮助别人。当我尝试包含18226行的2367KB csv文件时，不同php脚本花费的最少时间是（1） 来自php.net名为CsvImporter的fgetcsv文档，以及（2） file_get_contents=>PHP致命错误：允许的内存耗尽

（1） 取0.92574405670166（2） 取0.12543702125549（字符串形式）&0.52903485298157（拆分为阵列）注意：此计算不包括添加到mysql。

我找到的最佳解决方案使用3.0644409656525总计，包括添加到数据库和一些条件检查。处理一个8MB的文件花了11秒。解决方案是：

$csvInfo = analyse_file($file, 5);
    $lineSeperator = $csvInfo['line_ending']['value'];
    $fieldSeperator = $csvInfo['delimiter']['value'];
    $columns = getColumns($file);
    echo '<br>========Details========<br>';
    echo 'Line Sep: 't '.$lineSeperator;
    echo '<br>Field Sep:'t '.$fieldSeperator;
    echo '<br>Columns: ';print_r($columns);
    echo '<br>========Details========<br>';
    $ext = pathinfo($file, PATHINFO_EXTENSION);
    $table = str_replace(' ', '_', basename($file, "." . $ext));
    $rslt = table_insert($table, $columns);
    if($rslt){
        $query = "LOAD DATA LOCAL INFILE '".$file."' INTO TABLE $table FIELDS TERMINATED BY '$fieldSeperator' ";
        var_dump(addToDb($query, false));
    }

function addToDb($query, $getRec = true){
//echo '<br>Query : '.$query;
$con = @mysql_connect('localhost', 'root', '');
@mysql_select_db('rtest', $con);
$result = mysql_query($query, $con);
if($result){
    if($getRec){
         $data = array();
        while ($row = mysql_fetch_assoc($result)) { 
            $data[] = $row;
        }
        return $data;
    }else return true;
}else{
    var_dump(mysql_error());
    return false;
}
}

function table_insert($table_name, $table_columns) {
    $queryString = "CREATE TABLE " . $table_name . " (";
    $columns = '';
    $values = '';
    foreach ($table_columns as $column) {
        $values .= (strtolower(str_replace(' ', '_', $column))) . " VARCHAR(2048), ";
    }
    $values = substr($values, 0, strlen($values) - 2);
    $queryString .= $values . ") ";
    //// echo $queryString;
    return addToDb($queryString, false);
}

function getColumns($file){
    $cols = array();
    if (($handle = fopen($file, 'r')) !== FALSE)
    {
        while (($row = fgetcsv($handle)) !== FALSE) 
        {
           $cols = $row;
           if(count($cols)>0){
                break;
           }
        }
        return $cols;
    }else return false;
}
function analyse_file($file, $capture_limit_in_kb = 10) {
// capture starting memory usage
$output['peak_mem']['start']    = memory_get_peak_usage(true);
// log the limit how much of the file was sampled (in Kb)
$output['read_kb']                 = $capture_limit_in_kb;
// read in file
$fh = fopen($file, 'r');
    $contents = fread($fh, ($capture_limit_in_kb * 1024)); // in KB
fclose($fh);
// specify allowed field delimiters
$delimiters = array(
    'comma'     => ',',
    'semicolon' => ';',
    'tab'         => "'t",
    'pipe'         => '|',
    'colon'     => ':'
);
// specify allowed line endings
$line_endings = array(
    'rn'         => "'r'n",
    'n'         => "'n",
    'r'         => "'r",
    'nr'         => "'n'r"
);
// loop and count each line ending instance
foreach ($line_endings as $key => $value) {
    $line_result[$key] = substr_count($contents, $value);
}
// sort by largest array value
asort($line_result);
// log to output array
$output['line_ending']['results']     = $line_result;
$output['line_ending']['count']     = end($line_result);
$output['line_ending']['key']         = key($line_result);
$output['line_ending']['value']     = $line_endings[$output['line_ending']['key']];
$lines = explode($output['line_ending']['value'], $contents);
// remove last line of array, as this maybe incomplete?
array_pop($lines);
// create a string from the legal lines
$complete_lines = implode(' ', $lines);
// log statistics to output array
$output['lines']['count']     = count($lines);
$output['lines']['length']     = strlen($complete_lines);
// loop and count each delimiter instance
foreach ($delimiters as $delimiter_key => $delimiter) {
    $delimiter_result[$delimiter_key] = substr_count($complete_lines, $delimiter);
}
// sort by largest array value
asort($delimiter_result);
// log statistics to output array with largest counts as the value
$output['delimiter']['results']     = $delimiter_result;
$output['delimiter']['count']         = end($delimiter_result);
$output['delimiter']['key']         = key($delimiter_result);
$output['delimiter']['value']         = $delimiters[$output['delimiter']['key']];
// capture ending memory usage
$output['peak_mem']['end'] = memory_get_peak_usage(true);
return $output;
}

通常，当发生500错误时，会出现"服务不可用"错误。我认为这是因为执行时间不够。请检查您的日志/浏览器控制台，可能会看到500错误。

首先，保持set_time_limit（60）不在循环中。

做一些改变，比如

1. 对user_email_id列应用INDEX，这样您就可以使用select查询更快地获取行
2. 不要回显消息，保持输出缓冲区空闲

和

我使用开源程序完成了这些拍摄。你可以在这里买到http://sourceforge.net/projects/phpexcelreader/

试试这个。