在PHP中打开和关闭16000个MongoDB实例


Opening and closing 16000 Mongo DB instances in PHP

我目前正在进行一些个人测试和基准测试,以将MongoDB和MySQL的工作流程和效率与真实世界的示例数据进行比较。

为了在每个数据库中设置我的数据,我要进行数千次循环,并随机创建要插入数据库的数据对象。

然而,我在PHP中使用Mongo类时遇到了一些问题,无法解决。问题是这样的:

我有一个循环,它创建了一个新的Mongo实例和连接,在集合中插入一个小数组,然后关闭连接。这个循环应该运行20000次。然而,当它试图创建实例/建立连接时,它总是在第16300个循环附近失败(我想在运行几次后,最小值为16200,最大值为16350)。

循环中的代码如下:

$data = get_random_user_data();
$mongo = new Mongo('mongodb://admin:password@localhost:27017/test');
    if ($mongo->test->users->insert($data)) {
        $users[] = array('id' => $data['_id'], 'name' => $data['username']);
    echo $i." - Added user: ".$data['username'].'<br/>';
    }
$mongo->close();

get_random_user_data()只返回一个简单的关联数组。

我得到的错误是:

Fatal error: Uncaught exception 'MongoConnectionException' with message 'Unknown error'

在线:

$mongo = new Mongo('mongodb://admin:password@localhost:27017/test');

有什么想法吗?我是否缺少一些基本的东西,比如一些安全或垃圾邮件预防?

提前谢谢。

额外信息:

脚本在114.9797秒左右死亡。这不是一个基于PHP内存或时间的问题,因为所有的限制都提高了,昨天我运行了MySQL基准测试,在大约一个小时内插入了120000行(使用相同的循环打开连接、插入、关闭连接的方法),没有任何问题。

运行PHP 5.3.5版

phpinfoMongo-info:

MongoDB Support enabled
Version 1.2.0-
Directive   Local Value Master Value
mongo.allow_empty_keys  0   0
mongo.allow_persistent  1   1
mongo.auto_reconnect    1   1
mongo.chunk_size    262144  262144
mongo.cmd   $   $
mongo.default_host  localhost   localhost
mongo.default_port  27017   27017
mongo.long_as_object    0   0
mongo.native_long   0   0
mongo.no_id 0   0
mongo.utf8  1   1

您的操作系统愿意打开的套接字数量有限。当你打开一个套接字,然后关闭它时,操作系统不会立即将其放回"可用"池,它会在"时间等待"状态下挂一段时间,Nat在回答中提到了这一点。

您可以增加操作系统将打开的套接字数量,请参阅http://www.mongodb.org/display/DOCS/Too+许多+打开+文件(每个套接字都是一个打开的"文件")。

此外,如果您使用的是相当旧的驱动程序版本,您可能需要考虑升级。

我确认这也是ruby驱动程序的相同行为。我用ruby mongo复制了类似的情况,它通过每次打开/关闭mongo实例来插入20000条记录。在14110年到14200年间,它一直在失败。

这是错误消息

 uncaught throw #<Mongo::ConnectionFailure: Failed to connect to host localhost and port 27017: Cannot assign requested address - connect(2)>

这是完整的堆栈跟踪

Failed to connect to host localhost and port 27017: Cannot assign requested address - connect(2)
["/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:171:in `rescue in checkout_new_socket'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:166:in `checkout_new_socket'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:267:in `block (2 levels) in checkout'", "<internal:prelude>:10:in `synchronize'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:259:in `block in checkout'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:252:in `loop'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:252:in `checkout'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/connection.rb:496:in `checkout_writer'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/networking.rb:34:in `send_message'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/collection.rb:948:in `block in insert_documents'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/logging.rb:28:in `instrument'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/collection.rb:944:in `insert_documents'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/collection.rb:343:in `insert'", "/home/ramesh/Desktop/load_test.rb:15:in `block in load_test'", "/home/ramesh/Desktop/load_test.rb:6:in `times'", "/home/ramesh/Desktop/load_test.rb:6:in `load_test'", "(irb):2:in `irb_binding'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/workspace.rb:80:in `eval'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/workspace.rb:80:in `evaluate'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/context.rb:254:in `evaluate'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:159:in `block (2 levels) in eval_input'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:273:in `signal_status'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:156:in `block in eval_input'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:243:in `block (2 levels) in each_top_level_statement'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:229:in `loop'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:229:in `block in each_top_level_statement'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:228:in `catch'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:228:in `each_top_level_statement'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:155:in `eval_input'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:70:in `block in start'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:69:in `catch'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:69:in `start'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'"]
ArgumentError: uncaught throw #<Mongo::ConnectionFailure: Failed to connect to host localhost and port 27017: Cannot assign requested address - connect(2)>
    from /home/ramesh/Desktop/load_test.rb:21:in `throw'
    from /home/ramesh/Desktop/load_test.rb:21:in `rescue in block in load_test'
    from /home/ramesh/Desktop/load_test.rb:7:in `block in load_test'
    from /home/ramesh/Desktop/load_test.rb:6:in `times'
    from /home/ramesh/Desktop/load_test.rb:6:in `load_test'

正如@Chrisui所说,我的记忆力也没有下降。

这是我尝试过的脚本

require 'mongo'
def load_test
    start = Time.now
    puts 'starting at :' + start.to_s 
    20000.times do |i|    
            begin
                puts i      
            doc = {"name" => "MongoDB", "type" => "database", "count" => 1,"info" => {"x" => 203, "y" => '102'}}
            con = Mongo::Connection.new("localhost")
            db   = con['bulktest']
            coll = db['test']
            coll.insert(doc)
            con.close
                con,db,coll=nil,nil,nil
        rescue Exception => e  
            puts e.message  
            puts e.backtrace.inspect  
            throw e 
        end
    end
    stop = Time.now
    puts 'stoping at :' + stop.to_s 
    puts 'elapsed time is ' + (stop-start).to_s + ' seconds'
end

您的基准测试没有意义。不要保持打开/关闭连接。您应该重用持久化连接,而不是每次都打开/关闭。如果您快速打开和关闭套接字,您将有太多处于定时等待状态的套接字,这些套接字仍然使用文件描述符