Re,

Les services fonctionnent :

systemctl status waptserver.service
● waptserver.service - WAPT Server startup script
   Loaded: loaded (/usr/lib/systemd/system/waptserver.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-22 12:28:47 CEST; 1 day 21h ago
 Main PID: 1527 (python)
    Tasks: 1 (limit: 36864)
   CGroup: /system.slice/waptserver.service
           └─1527 /opt/wapt/bin/python /opt/wapt/waptserver/server.py

oct. 24 09:31:01 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:31:01,977 WARNING SocketIO connection refused for uuid 4C4C4544-004C-521
oct. 24 09:33:40 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:33:40,975 WARNING SocketIO connection refused for uuid 4C4C4544-004E-591
oct. 24 09:34:01 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:34:01,811 WARNING SocketIO connection refused for uuid 4C4C4544-0035-471
oct. 24 09:36:01 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:36:01,806 WARNING SocketIO connection refused for uuid 4C4C4544-0036-471
oct. 24 09:37:00 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:37:00,858 WARNING SocketIO connection refused for uuid 4C4C4544-004E-571
oct. 24 09:39:11 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:39:11,128 WARNING SocketIO connection refused for uuid 4C4C4544-0054-321
oct. 24 09:42:50 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:42:50,226 WARNING SocketIO connection refused for uuid 4C4C4544-0050-331
oct. 24 09:42:56 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:42:56,099 WARNING SocketIO connection refused for uuid 4C4C4544-0050-321
oct. 24 09:48:01 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:48:01,713 WARNING SocketIO connection refused for uuid 4C4C4544-0050-321
oct. 24 09:54:42 wapt2.insa-rennes.fr python[1527]: 2019-10-24 09:54:42,395 WARNING SocketIO connection refused for uuid 773F4CFF-233A-11E

systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/nginx.service.d
           └─nginx_worker_files_limit.conf
   Active: active (running) since Tue 2019-10-22 12:28:45 CEST; 1 day 21h ago
     Docs: man:nginx(8)
  Process: 1335 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 1293 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1340 (nginx)
    Tasks: 17 (limit: 36864)
   CGroup: /system.slice/nginx.service
           ├─1340 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─1341 nginx: worker process
           ├─1342 nginx: worker process
           ├─1343 nginx: worker process
           ├─1346 nginx: worker process
           ├─1347 nginx: worker process
           ├─1348 nginx: worker process
           ├─1349 nginx: worker process
           ├─1350 nginx: worker process
           ├─1351 nginx: worker process
           ├─1352 nginx: worker process
           ├─1354 nginx: worker process
           ├─1355 nginx: worker process
           ├─1356 nginx: worker process
           ├─1357 nginx: worker process
           ├─1358 nginx: worker process
           └─1359 nginx: worker process

oct. 22 12:28:45 wapt2.insa-rennes.fr systemd[1]: Starting A high performance web server and a reverse proxy server...
oct. 22 12:28:45 wapt2.insa-rennes.fr nginx[1293]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found
oct. 22 12:28:45 wapt2.insa-rennes.fr nginx[1335]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found
oct. 22 12:28:45 wapt2.insa-rennes.fr systemd[1]: Started A high performance web server and a reverse proxy server.

systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2019-10-22 12:28:47 CEST; 1 day 21h ago
  Process: 1525 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 1525 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 36864)
   CGroup: /system.slice/postgresql.service

oct. 22 12:28:47 wapt2.insa-rennes.fr systemd[1]: Starting PostgreSQL RDBMS...
oct. 22 12:28:47 wapt2.insa-rennes.fr systemd[1]: Started PostgreSQL RDBMS.

Le log fonctionne :

Oct 24 09:47:00 wapt2 systemd[32266]: Startup finished in 16ms.
Oct 24 09:47:00 wapt2 systemd[1]: Started User Manager for UID 0.
Oct 24 09:48:01 wapt2 python[1527]: 2019-10-24 09:48:01,713 WARNING SocketIO connection refused for uuid 4C4C4544-0050-3210-8058-B5C04F354B32, sid 549dcba1b526467aa16d4d396f942344: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:54:42 wapt2 python[1527]: 2019-10-24 09:54:42,395 WARNING SocketIO connection refused for uuid 773F4CFF-233A-11E4-8522-31D4970180FF, sid 83d5ea41409745cd9174de763e41a397: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:55:01 wapt2 CRON[44747]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 24 10:03:51 wapt2 systemd[1]: Started Run anacron jobs.
Oct 24 10:03:51 wapt2 anacron[44794]: Anacron 2.3 started on 2019-10-24
Oct 24 10:03:51 wapt2 anacron[44794]: Normal exit (0 jobs run)
Oct 24 10:03:51 wapt2 systemd[1]: anacron.timer: Adding 1.768916s random time.
Oct 24 10:05:01 wapt2 CRON[45080]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Que faire ?

Frédéric GARESCHÉ
Équipe Assistance et Exploitation
Direction du Système d'Information (D.S.I)
Correspondant Département EII et laboratoire IETR
Tél. : +33 (0)2 23 23 82 81
20 avenue des Buttes de Coësmes
CS 70839 - 35 708 RENNES Cedex 7




De: "Frederic Garesche" <frederic.garesche@insa-rennes.fr>
À: "Hubert TOUVET" <htouvet@tranquil.it>
Cc: "wapt" <wapt@lists.tranquil.it>
Envoyé: Jeudi 24 Octobre 2019 09:54:43
Objet: Arrêt aléatoire du serveur wapt

Bonjour,

Ça y est, le serveur est de nouveau tombé.

Voici la fin du syslog :

...
Oct 24 09:35:01 wapt2 CRON[32194]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 24 09:36:01 wapt2 python[1527]: 2019-10-24 09:36:01,806 WARNING SocketIO connection refused for uuid 4C4C4544-0036-4710-804D-C7C04F425832, sid d7dd626b8b1a4d00a164d4b8f130cd4b: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:37:00 wapt2 python[1527]: 2019-10-24 09:37:00,858 WARNING SocketIO connection refused for uuid 4C4C4544-004E-5710-8053-B5C04F354B32, sid 7fef72636b294b999db9bc9dd045ab4d: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:39:11 wapt2 python[1527]: 2019-10-24 09:39:11,128 WARNING SocketIO connection refused for uuid 4C4C4544-0054-3210-804D-CAC04F465931, sid a9ee20f9518645f596057db6c332d6a1: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:42:50 wapt2 python[1527]: 2019-10-24 09:42:50,226 WARNING SocketIO connection refused for uuid 4C4C4544-0050-3310-8057-B5C04F354B32, sid 1ed4b0741b224f869961b1d3e88c8702: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:42:56 wapt2 python[1527]: 2019-10-24 09:42:56,099 WARNING SocketIO connection refused for uuid 4C4C4544-0050-3210-8053-B5C04F354B32, sid bc178e93e17f4062b4fcc8fa77628242: SocketIO connection not authorized, invalid token: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand., instance
Oct 24 09:45:01 wapt2 CRON[32251]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 24 09:47:00 wapt2 systemd[1]: Created slice User Slice of root.
Oct 24 09:47:00 wapt2 systemd[1]: Starting User Manager for UID 0...
Oct 24 09:47:00 wapt2 systemd[1]: Started Session 341 of user root.
Oct 24 09:47:00 wapt2 systemd[1]: Started Session 342 of user root.
Oct 24 09:47:00 wapt2 systemd[1]: Started Session 344 of user root.
Oct 24 09:47:00 wapt2 systemd[32266]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Oct 24 09:47:00 wapt2 systemd[32266]: Reached target Paths.
Oct 24 09:47:00 wapt2 systemd[32266]: Listening on GnuPG cryptographic agent (access for web browsers).
Oct 24 09:47:00 wapt2 systemd[32266]: Listening on GnuPG cryptographic agent and passphrase cache.
Oct 24 09:47:00 wapt2 systemd[32266]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Oct 24 09:47:00 wapt2 systemd[32266]: Reached target Timers.
Oct 24 09:47:00 wapt2 systemd[32266]: Listening on GnuPG network certificate management daemon.
Oct 24 09:47:00 wapt2 systemd[32266]: Starting D-Bus User Message Bus Socket.
Oct 24 09:47:00 wapt2 systemd[32266]: Listening on D-Bus User Message Bus Socket.
Oct 24 09:47:00 wapt2 systemd[32266]: Reached target Sockets.
Oct 24 09:47:00 wapt2 systemd[32266]: Reached target Basic System.
Oct 24 09:47:00 wapt2 systemd[32266]: Reached target Default.
Oct 24 09:47:00 wapt2 systemd[32266]: Startup finished in 16ms.
Oct 24 09:47:00 wapt2 systemd[1]: Started User Manager for UID 0.

La commande pids=$(pgrep -f "server.py"); for f in /proc/$pids/fdinfo/*; do cat $f | grep flags | grep -q -v 400[0,1,2] && ls -al /proc/$pids/fd/$(basename $f); done; donne :
lr-x------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/0 -> /dev/null
lrwx------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/1 -> socket:[15786]
lr-x------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/15 -> /dev/urandom
lrwx------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/2 -> socket:[15786]
lrwx------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/3 -> anon_inode:[eventpoll]
lr-x------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/8 -> /dev/urandom

Cordialement,


Frédéric GARESCHÉ
Équipe Assistance et Exploitation
Direction du Système d'Information (D.S.I)
Correspondant Département EII et laboratoire IETR
Tél. : +33 (0)2 23 23 82 81
20 avenue des Buttes de Coësmes
CS 70839 - 35 708 RENNES Cedex 7




De: "Frederic Garesche" <frederic.garesche@insa-rennes.fr>
À: "Hubert TOUVET" <htouvet@tranquil.it>
Cc: "wapt" <wapt@lists.tranquil.it>
Envoyé: Mardi 22 Octobre 2019 15:53:08
Objet: Re: [Wapt] Arrêt aléatoire du serveur wapt

Ok, la commande fonctionne.
pids=$(pgrep -f "server.py"); for f in /proc/$pids/fdinfo/*; do cat $f | grep flags | grep -q -v 400[0,1,2] && ls -al /proc/$pids/fd/$(basename $f); done;
lr-x------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/0 -> /dev/null
lrwx------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/1 -> socket:[15786]
lr-x------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/15 -> /dev/urandom
lrwx------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/2 -> socket:[15786]
lrwx------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/3 -> anon_inode:[eventpoll]
lr-x------ 1 wapt www-data 64 oct.  22 12:28 /proc/1527/fd/8 -> /dev/urandom

Je n'ai plus qu'à attendre une panne...

Frédéric GARESCHÉ
Équipe Assistance et Exploitation
Direction du Système d'Information (D.S.I)
Correspondant Département EII et laboratoire IETR
Tél. : +33 (0)2 23 23 82 81
20 avenue des Buttes de Coësmes
CS 70839 - 35 708 RENNES Cedex 7