跟踪日志文件,并将结果写入新文件


Tailing Log File and Write results to new file

我不知道该怎么说,所以我把它打出来,然后编辑,然后回答任何出现的问题。

目前在我的本地网络设备上(基于PHP4),我使用这个来跟踪一个实时系统日志文件:http://commavee.com/2007/04/13/ajax-logfile-tailer-viewer/

这工作得很好,每1秒它加载一个外部页面(logfile.php)做一个tail -n 100 logfile.log脚本不做任何缓冲,所以它在屏幕上显示的结果是日志文件的最后100行。

logfile.php包含:

<? // logtail.php $cmd = "tail -10 /path/to/your/logs/some.log"; exec("$cmd 2>&1", $output);
foreach($output as $outputline) {
 echo ("$outputline'n");
}
?>

这部分工作正常。

我已经调整了logfile.php页面,将$outputline写入一个新的文本文件,只需使用fwrite($fp,$outputline."'n");

虽然这是有效的,但我在创建的新文件中有重复的问题。

显然,每次tail -n 100运行时都会产生结果,下一次运行时它可能会产生一些相同的行,由于这种重复,我可能会在新的文本文件中得到多行重复。

我不能直接比较我要写的行和前面的行,因为可能有相同的匹配。

是否有任何方法可以将当前的100行块与前一个块进行比较,然后只写不匹配的行…这也是阻碍A &B将包含所需的相同行…

是否可以更新logfile.php以注意它最后在我的日志文件中所处的位置,然后只从那里读取接下来的100行并将其写入新文件?

日志文件可能高达500MB,所以我不想每次都读取它。

欢迎提出任何意见或建议。

感谢

UPDATE @ 16:30

我使用:

$file = "/logs/syst.log";
$handle = fopen($file, "r");
if(isset($_SESSION['ftell'])) {   
    clearstatcache();
    fseek($handle, $_SESSION['ftell']); 
    while ($buffer = fgets($handle)) { 
        echo $buffer."<br/>";
        @ob_flush(); @flush();
    }   
    fclose($handle);
    @$_SESSION['ftell'] = ftell($handle);        
} else {
    fseek($handle, -1024, SEEK_END);
    fclose($handle);
     @$_SESSION['ftell'] = ftell($handle);
}

这似乎可以工作,但它首先加载整个文件,然后只加载更新。

我如何让它从最后50行开始,然后只更新?

谢谢:)

更新04/06/2013

我试过这段代码,它似乎更快,但它不只是从它离开的地方读取。

function last_lines($path, $line_count, $block_size = 512){
    $lines = array();
    // we will always have a fragment of a non-complete line
    // keep this in here till we have our next entire line.
    $leftover = "";
    $fh = fopen($path, 'r');
    // go to the end of the file
    fseek($fh, 0, SEEK_END);
    do{
        // need to know whether we can actually go back
        // $block_size bytes
        $can_read = $block_size;
        if(ftell($fh) < $block_size){
            $can_read = ftell($fh);
        }
        // go back as many bytes as we can
        // read them to $data and then move the file pointer
        // back to where we were.
        fseek($fh, -$can_read, SEEK_CUR);
        $data = fread($fh, $can_read);
        $data .= $leftover;
        fseek($fh, -$can_read, SEEK_CUR);
        // split lines by 'n. Then reverse them,
        // now the last line is most likely not a complete
        // line which is why we do not directly add it, but
        // append it to the data read the next time.
        $split_data = array_reverse(explode("'n", $data));
        $new_lines = array_slice($split_data, 0, -1);
        $lines = array_merge($lines, $new_lines);
        $leftover = $split_data[count($split_data) - 1];
    }
    while(count($lines) < $line_count && ftell($fh) != 0);
    if(ftell($fh) == 0){
        $lines[] = $leftover;
    }
    fclose($fh);
    // Usually, we will read too many lines, correct that here.
    return array_slice($lines, 0, $line_count);
}

任何方式都可以修改,以便它将从最后已知的位置读取。

?

谢谢

介绍

您可以通过跟踪文件的最后位置来跟踪文件;

$file = __DIR__ . "/a.log";
$tail = new TailLog($file);
$data = $tail->tail(100) ;
// Save $data to new file 

TailLog是我为这个任务写的一个简单的类,这里有一个简单的例子来显示它实际跟踪文件

简单测试

$file = __DIR__ . "/a.log";
$tail = new TailLog($file);
// Some Random Data
$data = array_chunk(range("a", "z"), 3);
// Write Log
file_put_contents($file, implode("'n", array_shift($data)));
// First Tail (2) Run
print_r($tail->tail(2));
// Run Tail (2) Again
print_r($tail->tail(2));
// Write Another data to Log
file_put_contents($file, "'n" . implode("'n", array_shift($data)), FILE_APPEND);
// Call Tail Again after writing Data
print_r($tail->tail(2));
// See the full content
print_r(file_get_contents($file));

// First Tail (2) Run
Array
(
    [0] => c
    [1] => b
)
// Run Tail (2) Again
Array
(
)
// Call Tail Again after writing Data
Array
(
    [0] => f
    [1] => e
)
// See the full content
a
b
c
d
e
f

实时跟踪

while(true) {
    $data = $tail->tail(100);
    // write data to another file
    sleep(5);
}

注意:尾随100行并不意味着总是返回100行。它会返回添加的新行100是返回的最大行数。如果有

,那么当日志记录超过每秒100行时,这可能效率不高。
<

尾类/strong>

class TailLog {
    private $file;
    private $data;
    private $timeout = 5;
    private $lock;
    function __construct($file) {
        $this->file = $file;
        $this->lock = new TailLock($file);
    }
    public function tail($lines) {
        $pos = - 2;
        $t = $lines;
        $fp = fopen($this->file, "r");
        $break = false;
        $line = "";
        $text = array();
        while($t > 0) {
            $c = "";
            // Seach for End of line
            while($c != "'n" && $c != PHP_EOL) {
                if (fseek($fp, $pos, SEEK_END) == - 1) {
                    $break = true;
                    break;
                }
                if (ftell($fp) < $this->lock->getPosition()) {
                    break;
                }
                $c = fgetc($fp);
                $pos --;
            }
            if (ftell($fp) < $this->lock->getPosition()) {
                break;
            }
            $t --;
            $break && rewind($fp);
            $text[$lines - $t - 1] = fgets($fp);
            if ($break) {
                break;
            }
        }
        // Move to end
        fseek($fp, 0, SEEK_END);
        // Save Position
        $this->lock->save(ftell($fp));
        // Close File
        fclose($fp);
        return array_map("trim", $text);
    }
}
<

尾巴锁/strong>

class TailLock {
    private $file;
    private $lock;
    private $data;
    function __construct($file) {
        $this->file = $file;
        $this->lock = $file . ".tail";
        touch($this->lock);
        if (! is_file($this->lock))
            throw new Exception("can't Create Lock File");
        $this->data = json_decode(file_get_contents($this->lock));
        // Check if file is valida json
        // Check if Data in the original files as not be delete
        // You expect data to increate not decrease
        if (! $this->data || $this->data->size > filesize($this->file)) {
            $this->reset($file);
        }
    }
    function getPosition() {
        return $this->data->position;
    }
    function reset() {
        $this->data = new stdClass();
        $this->data->size = filesize($this->file);
        $this->data->modification = filemtime($this->file);
        $this->data->position = 0;
        $this->update();
    }
    function save($pos) {
        $this->data = new stdClass();
        $this->data->size = filesize($this->file);
        $this->data->modification = filemtime($this->file);
        $this->data->position = $pos;
        $this->update();
    }
    function update() {
        return file_put_contents($this->lock, json_encode($this->data, 128));
    }
}

不太清楚如何使用输出,但会像这样工作....

$dat = file_get_contents("tracker.dat");
$fp = fopen("/logs/syst.log", "r");
fseek($fp, $dat, SEEK_SET);
ob_start();
// alternatively you can do a while fgets if you want to interpret the file or do something
fpassthru($fp);
$pos = ftell($fp);
fclose($fp);
echo nl2br(ob_get_clean());
file_put_contents("tracker.dat", ftell($fp));

tracker.dat只是一个文本文件,其中包含上次运行时读取位置的位置。我只是寻求那个位置和管道其余的输出缓冲区。

使用tail -c <number of bytes,而不是行数,然后检查文件大小。大致思路是:

$old_file_size = 0;
$max_bytes = 512;
function last_lines($path) {
  $new_file_size = filesize($path);
  $pending_bytes = $new_file_size - $old_file_size;
  if ($pending_bytes > $max_bytes) $pending_bytes = $max_bytes;
  exec("tail -c " + $pending_bytes + " /path/to/your_log", $output);
  $old_file_size = $new_file_size;
  return $output;
}

优点是你可以省去所有的特殊处理的东西,并获得良好的性能。缺点是您必须手动将输出分割成行,并且可能会以未完成的行结束。但这不是什么大问题,您可以通过从输出中单独省略最后一行(并适当地从old_file_size中减去最后一行的字节数)来轻松解决问题。