如何在不耗尽内存的情况下读取一个大的CSV文件


How can I read a big CSV file without Memory exhausted?

我有一个超过16 MB的CSV文件。

当我读到这篇文章时:

$exportString = @file_get_contents($url, false, stream_context_create($contextOptions)

我只是想在上做一个回应

$data=explode(';', $exportString);
echo $data[0];

然后这条消息出现在我的浏览器中:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 71 bytes) in....

在这之后,我想把它导入MySQL数据库。

有什么帮助吗?

php.ini文件中,您可以增加允许的内存大小

memory_limit = 512M

或者放在脚本的顶部:

ini_set("memory_limit","512M");

或者,如果您没有访问php.ini的权限,请在根目录中创建一个.htaccess文件,然后放入

php_value memory_limit = "512M"

编辑:268435456字节=256MB,所以请将其放大!

请记住,拥有巨大的内存限制并不能真正取代编写好的代码。最好使用file_get_contents附加参数offsetlength将其分割成块。


拆分不是一项琐碎的任务

但是,这里有一个关于如何做到这一点的简单算法

1. Initialize an empty string
  (begin a loop)
2. Grab a chunk from your file and append that to the string
3. Search for the last 'n character in that string (MAKE SURE IT ISN'T PART OF DATA)
  a. If 'n doesn't exist, continue
  b. If it does, grab the first substring up to that point and process that.
     Once finished grab the rest of substring assign it to your initial string.
  (loop until finished)
4. If there is data in the string left, do processing on that as well.

现在,算法找到最后一个"''n〃;字符串中

1. Initialize a variable called $inString = false and 
2. Initialize a variable $newLinePos = -1
3. Loop through each character of the string
  (begin loop)
  a. If the current charater is a double quote (")
     AND the character before IS NOT a backslash (')
     Then set $inString = !$inString;
  b. If $inString Then continue;
  c. If the current character is the newline ('n)
     Then set $newLinePos to the index of the current character
  (end loop)
4. If $newLinePos == -1 then we have not found any valid 'n and we need to grab more
   Otherwise, go on with the next part as perscribed above