php读取1Gb文件的方便内存限制应该是多少


What should be the convenient memory limit to php read 1Gb file?

我有一个必须读取的 1Gb 文件1。我选择 php 来过滤和更改一些行,然后使用这些更改创建另一个 file2。代码很好。如果我读取 50M 文件,它工作得很好。代码生成一个文件2,包括所有更改,如预期的那样。但是当我尝试运行 1Gb 文件时,没有创建 file2,并且我从浏览器收到如下错误消息:

The connection to localhost was interrupted.
Check your Internet connection
Check any cables and reboot any routers, modems, or other network devices you may be using.
Allow Chrome to access the network in your firewall or antivirus settings.
If it is already listed as a program allowed to access the network, try removing it from the list and adding it again.
If you use a proxy server...
Check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect "Use a proxy server for your LAN".

如果我返回并运行小文件,它再次运行良好。我已经将 php memeory 放在 2040M ini_set('memory_limit', '2048M')但我不知道它是否足够或是否可能。

那么,这个问题应该如何方便记忆呢?


注意:服务器是apache,win7,i7 8核64位,16G RAM。我认为代码并不重要,但有人要求查看它:

                       ini_set('memory_limit', '2048M');#Set the memory limit
                              $new_dbName="au_site";
                              $patern[0]="/di_site/";
                              $replace[0]=$new_dbName;
                              #xdebug_break();
                              $dirBase=dirname(__FILE__);
                              $dir2 = new DirectoryIterator(dirname(__FILE__));
                              #xdebug_break();
                              foreach ($dir2 as $fileinfo) {
                                     if (!$fileinfo->isDot() && $fileinfo->isFile()) {
                                          $str = $fileinfo->getFilename();
                                          if (preg_match("/'.sql/i", $str)) {
                                               #xdebug_break();
                                               $i=1;
                                               if(!($handle= fopen("$str", "r"))){
                                                die("Cannot open the file");
                                               }
                                               else{
                                                     while (!feof($handle)) {   
                                                             #xdebug_break();
                                                             $line=trim(fgets($handle), "'t'n'r'0'x0B");
                                                             $firstChar =  substr($line, 0,1) ;
                                                             $ord = ord($firstChar);
                                                             if(ord($firstChar)<>45){
                                                               if (preg_match("/di_site/", $line)) {
                                                                xdebug_break();
                                                                   $chn=preg_replace($patern, $replace, $line); 
                                                                   $line=$chn;
                                                               }
                                                               #echo $line."<br>";
                                                               $sql.=$line."'n";
                                                             }                                                            
                                                      }
                                                      xdebug_break();
                                                      $newDBsql=$dirBase."/".$new_dbName.".sql";
                                                      if(!$handle =  fopen($newDBsql,"w")){
                                                           die("Can not open the file");
                                                      }
                                                      else{
                                                       fwrite($handle, $sql);
                                                       fclose($handle);
                                                      }
                                               }
                                            } 
                                     }
                               } 

与其构建要写入的整个文件内容(这会占用大量内存),不如考虑编写流筛选器。

筛选器对来自基础流的单个缓冲读取进行操作,通常约为 8kB 的数据。以下示例代码定义了这样一个筛选器,它将每个存储桶拆分为单独的行,并调用您的代码对其进行更改。

<?php
class myfilter extends 'php_user_filter
{
  private $buffer; // internal buffer to create data buckets with
  private $pattern = ['/di_site/'];
  private $replace = ['au_site'];
  function filter($in, $out, &$consumed, $closing)
  {
    while ($bucket = stream_bucket_make_writeable($in)) {
      $parts = preg_split('/('n|'r'n)/', $bucket->data, -1, PREG_SPLIT_DELIM_CAPTURE);
      $buffer = '';
      // each line spans two array elements
      for ($i = 0, $n = count($parts); $i + 1 < $n; $i += 2) {
        $line = $parts[$i] . $parts[$i + 1];
        $buffer .= $this->treat_line($line);
        $consumed += strlen($line);
      }
      stream_bucket_append($out, stream_bucket_new($this->stream, $buffer));
    }
    return PSFS_PASS_ON;
  }
  /** THIS IS YOUR CODE **/
  function treat_line($line)
  {
    $line = trim($line, "'t'n'r'0'x0B");
    $firstChar =  substr($line, 0,1) ;
    if (ord($firstChar)<>45) {
      if (preg_match("/di_site/", $line)) {
        $line = preg_replace($this->pattern, $this->replace, $line);
      }
    }
    return $line . "'n";
  }
  function onCreate()
  {
    $this->buffer = fopen('php://memory', 'r+');
  }
  function onClose()
  {
    fclose($this->buffer);
  }
}
stream_filter_register("myfilter", "myfilter");
// open input and attach filter
$in = fopen(__FILE__, 'r');
stream_filter_prepend($in, 'myfilter');
// open output stream and start copying
$out = fopen('php://stdout', 'w');
stream_copy_to_stream($in, $out);
fclose($out);
fclose($in);