我使用php curl与nginx作为代理。下面是我的代码:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
之后的一段时间,运行nginx加载非常慢,有时它返回错误500。
日志显示
failed (24: Too many open files),
更多细节:
root@proxy-s2:~# ulimit -Hn
4096
root@proxy-s2:~# ulimit -Sn
1024
服务器上没有其他脚本在运行,也没有其他脚本在使用这个代理。
是nginx的bug吗?如何解决?
或
还能是什么呢?如何解决这个问题?
- 我没有改变nginx的默认配置
- Nginx重启解决了问题(暂时我猜)
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
server {
listen 8080;
location / {
resolver 8.8.8.8;
proxy_pass http://$http_host$uri$is_args$args;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
最高top - 09:23:55 up 21:51, 1 user, load average: 0.09, 0.13, 0.08
KiB Mem: 496164 total, 444328 used, 51836 free, 12300 buffers
KiB Swap: 0 total, 0 used, 0 free. 336228 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8 root 20 0 0 0 0 S 0.0 0.0 4:57.56 rcuos/0
4904 nobody 20 0 97796 14128 1012 R 0.0 2.8 4:19.82 nginx
7 root 20 0 0 0 0 S 0.0 0.0 2:11.35 rcu_sched
3 root 20 0 0 0 0 S 0.0 0.0 0:18.50 ksoftirqd/0
832 root 20 0 139208 6808 172 S 0.0 1.4 0:13.11 nova-agent
45 root 20 0 0 0 0 S 0.0 0.0 0:06.21 xenbus
74 root 20 0 0 0 0 S 0.0 0.0 0:03.03 kworker/u30:1
155 root 20 0 0 0 0 S 0.0 0.0 0:02.73 jbd2/xvda1-8
46 root 20 0 0 0 0 R 0.0 0.0 0:02.39 kworker/0:1
57 root 20 0 0 0 0 S 0.0 0.0 0:01.91 kswapd0
1 root 20 0 33448 2404 1136 S 0.0 0.5 0:01.47 init
391 root 20 0 18048 1336 996 S 0.0 0.3 0:00.97 xe-daemon
1034 syslog 20 0 255840 2632 784 S 0.0 0.5 0:00.90 rsyslogd
1107 root 20 0 61364 3048 2364 S 0.0 0.6 0:00.73 sshd
40 root rt 0 0 0 0 S 0.0 0.0 0:00.29 watchdog/0
316 root 20 0 19472 456 252 S 0.0 0.1 0:00.12 upstart-udev-br
6 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kworker/u30:0
1098 root 20 0 23652 1036 784 S 0.0 0.2 0:00.08 cron
7935 root 20 0 105632 4272 3284 S 0.0 0.9 0:00.07 sshd
330 root 20 0 51328 1348 696 S 0.0 0.3 0:00.06 systemd-udevd
7953 root 20 0 22548 3428 1680 S 0.0 0.7 0:00.05 bash
678 root 20 0 15256 524 268 S 0.0 0.1 0:00.04 upstart-socket-
8647 root 20 0 25064 1532 1076 R 0.0 0.3 0:00.03 top
mpstat
root@proxy-s2:~# mpstat
Linux 3.13.0-55-generic (proxy-s2) 07/09/2015 _x86_64_ (1 CPU)
09:22:17 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
09:22:17 AM all 0.94 0.00 1.63 0.16 0.00 2.16 0.92 0.00 0.00 94.20
iostat
root@proxy-s2:~# iostat
Linux 3.13.0-55-generic (proxy-s2) 07/09/2015 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.94 0.00 3.80 0.16 0.92 94.19
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvdc 0.01 0.02 0.00 1710 0
xvda 3.16 4.19 88.56 322833 6815612
请在下面尝试,在您的limits.conf中做以下更改。
vi/etc/security/limits.conf
对于打开的文件
- 软文件64000
-
硬文件64000
对于最大用户进程
-
soft nproc 47758
-
hard nproc 47758
为最大内存大小
-
软rss无限
-
硬rss无限
虚拟内存
-
soft as unlimited
- hard as unlimited
把这个放到Nginx配置文件的顶部:
worker_rlimit_nofile 40000;
events {
worker_connections 4096;
}
我想我找到问题了:
这里是nginx error.log
2015/07/09 14:17:27 [error] 15390#0: *7549 connect() failed (111: Connection refused) while connecting to upstream, client: 23.239.194.233, server: , request: "GET http://www.lgqfz.com/ HTTP/1.1", upstream: "http://127.0.0.3:80/", host: "www.lgqfz.com", referrer: "http://www.baidu.com"
2015/07/09 14:17:29 [error] 15390#0: *8121 connect() failed (111: Connection refused) while connecting to upstream, client: 204.44.65.119, server: , request: "GET http://www.lgqfz.com/ HTTP/1.1", upstream: "http://127.0.0.3:80/", host: "www.lgqfz.com", referrer: "http://www.baidu.com"
2015/07/09 14:17:32 [error] 15390#0: *8650 connect() failed (101: Network is unreachable) while connecting to upstream, client: 78.47.53.98, server: , request: "GET http://188.8.253.161/ HTTP/1.1", upstream: "http://188.8.253.161:80/", host: "188.8.253.161", referrer: "http://188.8.253.161/"
这是对我的代理的DDOS攻击,我只允许我的IP访问代理。
我发现这是很常见的最近-当你抓取一个网站,网站识别你作为一个爬虫,它有时会DDOS攻击你的代理,直到他们变黑。其中一个例子是amazon.com