⚓ T370304 Bursts of occasional severe contention on s4 (commonswiki) primary mariadb causing recurrent user-facing outages on all wikis
Article Images
Bursts of occasional severe contention on s4 (commonswiki) primary mariadb causing recurrent user-facing outages on all wikis
Closed, ResolvedPublicSecurity
Bursts of occasional severe contention on s4 (commonswiki) primary mariadb causing recurrent user-facing outages on all wikis
Closed, ResolvedPublicSecurity
Steps to replicate the issue (include links if applicable):
- Do stuff a lot
- Maybe it'll happen
What happens?:
On July 13, I got this:
MediaWiki internal error.
Original exception: [c8466b8a-0c54-4456-8259-2e3fd3f4be95] 2024-07-13 19:12:17: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"
Exception caught inside exception handler.
Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.
I just got another one
MediaWiki internal error.
Original exception: [38973e56-81e9-4748-81c2-6735336e3a90] 2024-07-17 15:51:37: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"
Exception caught inside exception handler.
Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.
URL for today's was: https://en.wikipedia.org/w/index.php?title=Wikipedia:Arbitration/Requests/Case&curid=22747298&diff=1235079535&oldid=1235079140
I figure if I got two of them in a few days, it might be something significant, so logging this report.
Error
- mwversion: 1.43.0-wmf.13
- reqId: 38973e56-81e9-4748-81c2-6735336e3a90
- Find reqId in Logstash
normalized_message
[{reqId}] {exception_url} Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded. In order to protect application servers, the circuit breaking to databases of this section have been activated. Please try again a few seconds.
exception.trace
from /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/loadmonitor/LoadMonitor.php(124) #0 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(455): Wikimedia\Rdbms\LoadMonitor->scaleLoads(array) #1 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(781): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string) #2 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/database/DBConnRef.php(111): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer) #3 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/database/DBConnRef.php(125): Wikimedia\Rdbms\DBConnRef->ensureConnection() #4 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/database/DBConnRef.php(351): Wikimedia\Rdbms\DBConnRef->__call(string, array) #5 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(746): Wikimedia\Rdbms\DBConnRef->select(array, array, array, string, array, array) #6 /srv/mediawiki/php-1.43.0-wmf.13/extensions/Echo/includes/Mapper/EventMapper.php(177): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet() #7 /srv/mediawiki/php-1.43.0-wmf.13/extensions/Echo/includes/Hooks.php(879): MediaWiki\Extension\Notifications\Mapper\EventMapper->fetchUnreadByUserAndPage(MediaWiki\User\User, integer) #8 /srv/mediawiki/php-1.43.0-wmf.13/extensions/Echo/includes/Hooks.php(979): MediaWiki\Extension\Notifications\Hooks->processMarkAsRead(MediaWiki\User\User, MediaWiki\Request\WebRequest, MediaWiki\Title\Title) #9 /srv/mediawiki/php-1.43.0-wmf.13/includes/HookContainer/HookContainer.php(159): MediaWiki\Extension\Notifications\Hooks->onSkinTemplateNavigation__Universal(MediaWiki\Skins\Vector\SkinVectorLegacy, array) #10 /srv/mediawiki/php-1.43.0-wmf.13/includes/HookContainer/HookRunner.php(3591): MediaWiki\HookContainer\HookContainer->run(string, array, array) #11 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(969): MediaWiki\HookContainer\HookRunner->onSkinTemplateNavigation__Universal(MediaWiki\Skins\Vector\SkinVectorLegacy, array) #12 /srv/mediawiki/php-1.43.0-wmf.13/skins/Vector/includes/SkinVectorLegacy.php(39): SkinTemplate->runOnSkinTemplateNavigationHooks(MediaWiki\Skins\Vector\SkinVectorLegacy, array) #13 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(1361): MediaWiki\Skins\Vector\SkinVectorLegacy->runOnSkinTemplateNavigationHooks(MediaWiki\Skins\Vector\SkinVectorLegacy, array) #14 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(609): SkinTemplate->buildContentNavigationUrlsInternal() #15 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(185): SkinTemplate->getPortletsTemplateData() #16 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinMustache.php(139): SkinTemplate->getTemplateData() #17 /srv/mediawiki/php-1.43.0-wmf.13/skins/Vector/includes/SkinVectorLegacy.php(173): SkinMustache->getTemplateData() #18 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinMustache.php(92): MediaWiki\Skins\Vector\SkinVectorLegacy->getTemplateData() #19 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(178): SkinMustache->generateHTML() #20 /srv/mediawiki/php-1.43.0-wmf.13/includes/Output/OutputPage.php(3027): SkinTemplate->outputPage() #21 /srv/mediawiki/php-1.43.0-wmf.13/includes/actions/ActionEntryPoint.php(162): MediaWiki\Output\OutputPage->output(boolean) #22 /srv/mediawiki/php-1.43.0-wmf.13/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute() #23 /srv/mediawiki/php-1.43.0-wmf.13/index.php(58): MediaWiki\MediaWikiEntryPoint->run() #24 /srv/mediawiki/w/index.php(3): require(string) #25 {main}
Impact
Notes
- Request URL
- https://en.wikipedia.org/w/index.php?curid=*&diff=*&oldid=*&title=*
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
I am making this private for now, as it is a DDoS vector - we still don't know what it is, but this brings down many wikis. Just being careful here.
It seems that, the initial linter changes were likely just one part of the overall effort to stabilize the system. Once this is resolved, we should consider re-enabling the linting rules to ensure they are unrelated, unless you see a risk on doing it. Thoughts?
This just happened again.
I am not posting the list of queries as it can be just a red herring, but I am going to be posting the list of IPs that were making the write requests that got stuck at around 14:22
1 | 10.67.177.154:33858 |
---|---|
2 | 10.67.188.105:60518 |
3 | 10.67.137.180:49490 |
4 | 10.67.154.85:42378 |
5 | 10.67.160.43:39320 |
6 | 10.67.178.150:58012 |
7 | 10.67.186.61:38074 |
8 | 10.67.144.24:41676 |
9 | 10.67.183.152:59672 |
10 | 10.67.150.252:36458 |
11 | 10.67.152.124:48928 |
12 | 10.67.138.80:55714 |
13 | 10.67.168.171:51018 |
14 | 10.67.160.160:53440 |
15 | 10.67.164.46:56220 |
16 | 10.67.145.165:48656 |
17 | 10.67.187.26:39474 |
18 | 10.67.137.138:58128 |
19 | 10.67.134.128:32964 |
20 | 10.67.144.175:50296 |
21 | 10.67.136.183:46752 |
22 | 10.67.162.170:37452 |
23 | 10.67.152.212:45720 |
24 | 10.67.144.49:54582 |
25 | 10.67.131.216:48590 |
26 | 10.67.189.228:38632 |
27 | 10.67.162.170:37460 |
28 | 10.67.175.62:58774 |
29 | 10.67.151.209:36316 |
30 | 10.67.170.139:45322 |
31 | 10.67.174.124:46170 |
32 | 10.67.188.73:53434 |
33 | 10.67.139.73:41586 |
34 | 10.67.184.55:42402 |
35 | 10.67.144.215:38262 |
36 | 10.67.128.226:41434 |
37 | 10.67.152.88:53708 |
38 | 10.67.175.62:58790 |
39 | 10.67.144.205:57258 |
40 | 10.67.158.44:57956 |
41 | 10.67.176.115:37680 |
42 | 10.67.138.217:50578 |
43 | 10.67.171.204:53446 |
44 | 10.67.163.124:38264 |
45 | 10.67.159.90:51012 |
46 | 10.67.163.124:38280 |
47 | 10.67.151.209:36326 |
48 | 10.67.160.151:60196 |
49 | 10.67.187.139:39678 |
50 | 10.67.183.26:40690 |
51 | 10.67.144.205:57268 |
52 | 10.67.145.35:47290 |
53 | 10.67.135.96:55340 |
54 | 10.67.161.243:45694 |
55 | 10.67.145.165:48666 |
56 | 10.67.188.228:39820 |
57 | 10.67.133.94:53062 |
58 | 10.67.183.152:59698 |
59 | 10.67.175.62:58802 |
60 | 10.67.175.62:58812 |
61 | 10.67.190.169:53470 |
62 | 10.67.174.60:38298 |
63 | 10.67.169.73:41694 |
64 | 10.67.163.124:38306 |
65 | 10.67.154.85:33610 |
66 | 10.67.135.12:56654 |
67 | 10.67.140.87:40096 |
68 | 10.67.174.206:42570 |
69 | 10.67.132.4:52032 |
70 | 10.67.186.233:60396 |
71 | 10.67.186.184:45258 |
72 | 10.67.152.88:51480 |
73 | 10.67.152.20:34872 |
74 | 10.67.157.35:42414 |
75 | 10.67.185.93:40214 |
76 | 10.67.140.139:47546 |
77 | 10.67.147.135:52784 |
78 | 10.67.181.221:38686 |
79 | 10.67.133.94:37738 |
80 | 10.67.154.85:33630 |
81 | 10.67.184.149:42828 |
82 | 10.67.145.255:50734 |
83 | 10.67.134.60:34744 |
84 | 10.67.158.230:46494 |
85 | 10.67.130.8:33012 |
86 | 10.67.180.161:42548 |
87 | 10.67.141.54:34236 |
88 | 10.67.141.226:48748 |
89 | 10.67.153.68:46278 |
90 | 10.67.157.86:56312 |
91 | 10.67.164.46:56230 |
92 | 10.67.187.26:39486 |
93 | 10.67.156.251:49698 |
94 | 10.67.170.24:58002 |
95 | 10.67.180.134:57156 |
96 | 10.67.157.86:56314 |
97 | 10.67.150.227:57714 |
98 | 10.67.134.128:32994 |
99 | 10.67.188.228:39836 |
100 | 10.67.181.106:33750 |
101 | 10.67.139.73:41620 |
102 | 10.67.144.231:59940 |
103 | 10.67.190.233:44340 |
104 | 10.67.135.12:56668 |
105 | 10.67.152.212:45760 |
106 | 10.67.183.152:53302 |
107 | 10.67.153.223:39790 |
108 | 10.67.158.225:41218 |
109 | 10.67.135.96:55362 |
110 | 10.67.176.36:51434 |
111 | 10.67.132.22:42118 |
112 | 10.67.187.216:52396 |
113 | 10.67.142.12:44418 |
114 | 10.67.188.249:51312 |
115 | 10.67.164.81:58546 |
116 | 10.67.160.206:37982 |
117 | 10.67.180.40:43678 |
118 | 10.67.134.60:34756 |
119 | 10.67.170.139:48122 |
120 | 10.67.186.61:38128 |
121 | 10.67.179.223:32990 |
122 | 10.67.180.134:57160 |
123 | 10.67.164.81:58562 |
124 | 10.67.170.139:48130 |
125 | 10.67.138.217:43858 |
126 | 10.67.185.230:34802 |
127 | 10.67.178.102:54706 |
128 | 10.67.145.165:39626 |
129 | 10.67.168.79:50678 |
130 | 10.67.142.64:42612 |
131 | 10.67.186.233:60408 |
132 | 10.67.149.140:55800 |
133 | 10.67.134.60:34758 |
134 | 10.67.145.44:50654 |
135 | 10.67.145.71:58668 |
136 | 10.67.130.8:33028 |
137 | 10.67.158.47:39940 |
138 | 10.67.146.71:35634 |
139 | 10.67.151.209:37066 |
140 | 10.67.174.124:36576 |
141 | 10.67.179.223:33002 |
142 | 10.67.160.206:37996 |
143 | 10.67.130.149:58920 |
144 | 10.67.163.138:51926 |
145 | 10.67.146.71:35650 |
146 | 10.67.165.218:57954 |
147 | 10.67.160.2:43464 |
148 | 10.67.174.119:54768 |
149 | 10.67.130.241:55386 |
150 | 10.67.186.61:38138 |
151 | 10.67.140.87:40102 |
152 | 10.67.136.182:45342 |
153 | 10.67.160.160:56430 |
154 | 10.67.136.182:45344 |
155 | 10.67.150.50:59202 |
156 | 10.67.180.134:57170 |
157 | 10.67.157.35:42422 |
158 | 10.67.137.180:59162 |
159 | 10.67.152.88:51488 |
160 | 10.67.169.216:41968 |
161 | 10.67.131.216:48302 |
162 | 10.67.152.88:51498 |
163 | 10.67.132.121:53176 |
164 | 10.67.138.251:55082 |
165 | 10.67.140.139:58436 |
166 | 10.67.142.158:49020 |
167 | 10.67.161.243:45716 |
168 | 10.67.134.128:32996 |
169 | 10.67.144.231:59946 |
170 | 10.67.128.226:52226 |
171 | 10.67.150.50:59212 |
172 | 10.67.140.139:58444 |
173 | 10.67.141.54:34244 |
174 | 10.67.145.165:39634 |
175 | 10.67.167.12:36370 |
176 | 10.67.163.79:59184 |
177 | 10.67.148.118:53090 |
178 | 10.67.186.233:60418 |
179 | 10.67.161.243:45718 |
180 | 10.67.145.66:34300 |
181 | 10.67.135.12:56678 |
182 | 10.67.150.27:34494 |
183 | 10.67.183.152:53304 |
184 | 10.67.131.29:40518 |
185 | 10.67.183.192:45408 |
186 | 10.67.163.249:53556 |
187 | 10.67.183.26:49270 |
188 | 10.67.130.8:52346 |
189 | 10.67.186.184:58798 |
190 | 10.67.165.248:43536 |
191 | 10.67.129.137:45724 |
192 | 10.67.156.251:57192 |
193 | 10.67.183.26:49272 |
194 | 10.67.158.225:41230 |
195 | 10.67.141.235:51486 |
196 | 10.67.135.12:56686 |
197 | 10.67.169.216:41982 |
198 | 10.67.181.106:33764 |
199 | 10.67.164.46:56240 |
200 | 10.67.136.98:49288 |
201 | 10.67.153.18:49864 |
202 | 10.67.143.217:52162 |
203 | 10.67.188.11:44772 |
204 | 10.67.141.119:55206 |
205 | 10.67.136.183:59040 |
206 | 10.67.133.94:37744 |
207 | 10.67.174.124:36586 |
208 | 10.67.145.35:55828 |
209 | 10.67.178.102:54718 |
210 | 10.67.187.26:39506 |
211 | 10.67.146.207:34470 |
212 | 10.67.141.119:55208 |
213 | 10.67.141.193:41780 |
214 | 10.67.164.46:56244 |
215 | 10.67.174.60:38304 |
216 | 10.67.163.124:38314 |
217 | 10.67.152.148:42700 |
218 | 10.67.132.121:53188 |
219 | 10.67.171.204:53456 |
220 | 10.67.163.138:35460 |
221 | 10.67.133.158:40982 |
222 | 10.67.131.216:48304 |
223 | 10.67.129.103:60896 |
224 | 10.67.165.205:39850 |
225 | 10.67.138.48:58740 |
226 | 10.67.181.106:42000 |
227 | 10.67.161.243:50846 |
228 | 10.67.190.233:45610 |
229 | 10.67.186.184:58804 |
230 | 10.67.132.88:36040 |
231 | 10.67.148.100:43774 |
232 | 10.67.137.143:35716 |
233 | 10.67.152.219:35926 |
234 | 10.67.142.143:55376 |
235 | 10.67.134.23:43270 |
236 | 10.67.178.102:54722 |
237 | 10.67.147.132:42194 |
238 | 10.67.190.232:45964 |
239 | 10.67.158.47:41344 |
240 | 10.67.168.40:43950 |
241 | 10.67.180.40:43692 |
242 | 10.67.151.53:60708 |
243 | 10.67.178.150:57052 |
244 | 10.67.148.100:43786 |
245 | 10.67.191.68:41928 |
246 | 10.67.145.66:33082 |
247 | 10.67.183.26:49280 |
248 | 10.67.181.221:49196 |
249 | 10.67.188.25:57910 |
250 | 10.67.138.217:43872 |
251 | 10.67.150.15:52360 |
252 | 10.67.179.223:33014 |
253 | 10.67.153.223:39794 |
254 | 10.67.145.66:33088 |
255 | 10.67.129.103:60912 |
256 | 10.67.188.105:57598 |
257 | 10.67.128.226:52236 |
258 | 10.67.135.181:50854 |
259 | 10.67.161.243:50852 |
260 | 10.67.170.24:58010 |
261 | 10.67.150.227:57720 |
262 | 10.67.170.139:48142 |
263 | 10.67.191.68:41942 |
264 | 10.67.145.174:53940 |
265 | 10.67.161.42:57488 |
266 | 10.67.174.124:36602 |
267 | 10.67.138.217:43886 |
268 | 10.67.160.160:56438 |
269 | 10.67.163.79:56444 |
270 | 10.67.181.138:57174 |
271 | 10.67.128.219:56398 |
272 | 10.67.183.26:49288 |
273 | 10.67.173.239:55852 |
274 | 10.67.131.216:48306 |
275 | 10.67.163.138:35488 |
276 | 10.67.136.183:59042 |
277 | 10.67.128.226:52244 |
278 | 10.67.145.66:33096 |
279 | 10.67.184.55:34588 |
280 | 10.67.139.227:47314 |
281 | 10.67.188.105:57604 |
282 | 10.67.189.251:44490 |
283 | 10.67.150.27:34506 |
284 | 10.67.163.236:35280 |
285 | 10.67.138.80:55766 |
286 | 10.67.140.139:58450 |
287 | 10.67.163.124:49684 |
288 | 10.67.157.35:42434 |
289 | 10.67.136.182:45358 |
290 | 10.67.186.233:39718 |
291 | 10.67.189.251:44494 |
292 | 10.67.152.219:35938 |
293 | 10.67.150.227:57730 |
294 | 10.67.138.251:55084 |
295 | 10.67.152.212:50924 |
296 | 10.67.145.71:49780 |
297 | 10.67.146.144:56840 |
298 | 10.67.147.21:33110 |
299 | 10.67.138.110:49526 |
300 | 10.67.189.208:55356 |
301 | 10.67.132.121:49554 |
302 | 10.67.190.233:45614 |
303 | 10.67.185.230:34812 |
304 | 10.67.135.12:43380 |
305 | 10.67.131.124:46704 |
306 | 10.67.140.215:53468 |
307 | 10.67.141.226:37966 |
308 | 10.67.166.189:49562 |
309 | 10.67.163.79:56476 |
310 | 10.67.140.215:53472 |
311 | 10.67.186.61:39304 |
312 | 10.67.167.72:45090 |
313 | 10.67.132.33:59846 |
314 | 10.67.140.215:53488 |
315 | 10.67.180.153:39458 |
316 | 10.67.134.74:47552 |
317 | 10.67.163.168:38716 |
318 | 10.67.145.172:45428 |
319 | 10.67.162.3:33550 |
320 | 10.67.181.115:45002 |
321 | 10.67.164.144:50428 |
322 | 10.67.151.53:37836 |
323 | 10.67.133.215:59836 |
324 | 10.67.154.25:50786 |
325 | 10.67.134.159:46270 |
326 | 10.67.133.109:45876 |
327 | 10.67.143.182:40202 |
328 | 10.67.157.110:41112 |
329 | 10.67.146.144:56856 |
330 | 10.67.133.158:42526 |
331 | 10.67.139.225:47912 |
332 | 10.67.163.249:36750 |
333 | 10.67.136.112:48324 |
334 | 10.67.168.40:54506 |
335 | 10.67.183.70:58178 |
336 | 10.67.174.173:46092 |
337 | 10.67.190.8:39486 |
338 | 10.67.145.35:40110 |
339 | 10.67.160.43:56208 |
340 | 10.67.133.158:42542 |
341 | 10.67.174.167:58880 |
342 | 10.67.133.243:45816 |
343 | 10.67.174.39:48564 |
344 | 10.67.184.190:36416 |
345 | 10.67.146.198:56692 |
346 | 10.67.157.20:56198 |
347 | 10.67.190.204:50570 |
348 | 10.67.131.120:32964 |
349 | 10.67.164.144:45810 |
350 | 10.67.152.133:38296 |
351 | 10.67.144.135:44654 |
352 | 10.67.162.145:47206 |
353 | 10.67.150.27:44636 |
354 | 10.67.163.254:38854 |
355 | 10.67.187.182:36390 |
356 | 10.67.184.14:57040 |
357 | 10.67.131.124:45772 |
358 | 10.67.131.250:56976 |
359 | 10.67.188.65:60806 |
360 | 10.67.172.222:59796 |
361 | 10.67.188.25:37940 |
362 | 10.67.168.118:58570 |
363 | 10.67.172.29:37596 |
364 | 10.67.163.179:57346 |
365 | 10.67.143.240:51480 |
366 | 10.67.166.153:41740 |
367 | 10.67.137.180:52066 |
368 | 10.67.146.198:56694 |
369 | 10.67.138.115:58038 |
370 | 10.67.148.118:41202 |
371 | 10.67.149.98:53568 |
372 | 10.67.137.233:53912 |
373 | 10.67.178.147:54352 |
374 | 10.67.145.174:36806 |
375 | 10.67.167.78:46864 |
376 | 10.67.170.172:37406 |
377 | 10.67.171.204:41324 |
378 | 10.67.145.44:56820 |
379 | 10.67.168.14:60770 |
380 | 10.67.136.73:32980 |
381 | 10.67.163.79:38694 |
382 | 10.67.168.40:48856 |
383 | 10.67.159.127:33180 |
384 | 10.67.190.232:35498 |
385 | 10.67.151.50:51720 |
386 | 10.67.144.208:57882 |
387 | 10.67.148.26:58038 |
388 | 10.67.188.82:41268 |
389 | 10.67.148.118:59578 |
390 | 10.67.138.61:38090 |
391 | 10.67.174.77:54870 |
392 | 10.67.130.151:47846 |
393 | 10.67.165.248:51896 |
394 | 10.67.181.201:34864 |
395 | 10.67.139.70:52604 |
396 | 10.67.188.105:36708 |
397 | 10.67.131.11:57276 |
398 | 10.67.184.176:45200 |
399 | 10.67.187.242:45948 |
400 | 10.67.162.145:55396 |
401 | 10.67.152.204:53646 |
402 | 10.67.142.143:41386 |
403 | 10.67.168.130:57092 |
404 | 10.67.142.143:41400 |
405 | 10.67.146.89:59324 |
406 | 10.67.134.12:58962 |
407 | 10.67.152.41:41294 |
408 | 10.67.164.144:36478 |
409 | 10.67.142.218:50152 |
410 | 10.67.158.38:58656 |
411 | 10.67.152.171:42736 |
412 | 10.67.153.196:38402 |
413 | 10.67.188.25:37954 |
414 | 10.67.182.175:54562 |
415 | 10.67.158.38:56102 |
416 | 10.67.154.25:36120 |
417 | 10.67.176.9:51434 |
418 | 10.67.164.127:33108 |
419 | 10.67.153.115:48544 |
420 | 10.67.181.138:57138 |
421 | 10.67.164.127:33120 |
422 | 10.67.162.3:43092 |
423 | 10.67.145.222:60590 |
424 | 10.67.145.172:60008 |
425 | 10.67.143.99:49034 |
426 | 10.67.153.243:45700 |
427 | 10.67.154.48:37698 |
428 | 10.67.164.18:51036 |
429 | 10.67.135.96:48728 |
430 | 10.67.147.188:57196 |
431 | 10.67.151.182:35634 |
432 | 10.67.146.71:41548 |
433 | 10.67.188.232:60202 |
434 | 10.67.185.93:55736 |
435 | 10.67.129.15:60506 |
436 | 10.67.130.225:56272 |
437 | 10.67.140.215:54274 |
438 | 10.67.142.64:39708 |
439 | 10.67.143.217:47518 |
440 | 10.67.136.135:48228 |
441 | 10.67.160.43:55858 |
442 | 10.67.176.164:48702 |
443 | 10.67.152.148:46512 |
444 | 10.64.48.191:46450 |
445 | 10.67.143.197:34854 |
446 | 10.67.154.28:60948 |
447 | 10.67.167.12:38636 |
448 | 10.67.134.159:50926 |
449 | 10.67.146.71:50684 |
450 | 10.67.147.150:55108 |
451 | 10.67.184.143:45316 |
452 | 10.67.158.195:47708 |
453 | 10.67.158.195:47710 |
454 | 10.67.138.217:44052 |
455 | 10.67.179.245:42322 |
456 | 10.67.156.202:49036 |
457 | 10.67.160.206:53110 |
458 | 10.67.132.107:48058 |
459 | 10.67.138.251:44502 |
460 | 10.67.179.251:38946 |
461 | 10.67.157.112:57274 |
462 | 10.67.133.194:50222 |
463 | 10.67.172.30:45584 |
464 | 10.67.181.241:35520 |
465 | 10.67.180.40:51006 |
466 | 10.67.154.90:32918 |
467 | 10.67.139.35:55798 |
468 | 10.67.162.240:58714 |
469 | 10.67.152.219:49846 |
470 | 10.67.140.215:55044 |
471 | 10.67.133.247:37758 |
472 | 10.67.137.138:41646 |
473 | 10.67.177.130:48982 |
474 | 10.67.133.93:51766 |
475 | 10.67.162.191:46942 |
476 | 10.67.174.154:54776 |
477 | 10.67.162.170:39040 |
478 | 10.67.190.232:54118 |
479 | 10.67.136.135:50950 |
480 | 10.67.174.60:55392 |
481 | 10.67.132.22:49308 |
482 | 10.67.142.69:60028 |
483 | 10.67.181.106:35076 |
484 | 10.67.139.58:44016 |
485 | 10.194.182.41:45658 |
486 | 10.67.183.9:34526 |
487 | 10.67.182.132:57146 |
488 | 10.67.148.50:54750 |
489 | 10.67.131.29:56224 |
490 | 10.67.132.107:43370 |
491 | 10.67.183.9:39670 |
492 | 10.67.158.207:35464 |
493 | 10.67.148.216:33134 |
494 | 10.67.187.65:56480 |
495 | 10.67.170.161:34030 |
496 | 10.67.188.229:45858 |
497 | 10.67.133.243:45214 |
498 | 10.67.148.15:58384 |
499 | 10.67.135.181:36870 |
500 | 10.67.158.38:52686 |
501 | 10.67.133.158:43302 |
502 | 10.67.159.147:48990 |
503 | 10.67.183.26:59334 |
504 | 10.67.152.171:44720 |
505 | 10.67.188.105:47904 |
506 | 10.67.143.197:41728 |
507 | 10.67.180.193:33216 |
508 | 10.67.130.190:49500 |
509 | 10.67.131.124:55112 |
510 | 10.67.159.213:49644 |
511 | 10.67.182.175:45620 |
512 | 10.67.140.77:43572 |
513 | 10.67.163.254:38878 |
514 | 10.67.187.182:49094 |
515 | 10.67.166.189:45350 |
516 | 10.67.138.61:51074 |
517 | 10.67.162.206:53214 |
518 | 10.67.190.232:60340 |
519 | 10.67.183.153:50632 |
520 | 10.67.186.131:38476 |
521 | 10.67.172.168:55094 |
522 | 10.67.176.85:58434 |
523 | 10.67.180.193:39482 |
524 | 10.67.138.110:58076 |
525 | 10.67.138.110:58082 |
526 | 10.67.142.233:50172 |
527 | 10.67.134.12:54202 |
528 | 10.67.148.118:49842 |
529 | 10.67.191.119:47894 |
530 | 10.67.184.9:38320 |
531 | 10.67.173.239:38304 |
532 | 10.67.130.190:44210 |
533 | 10.67.159.40:46864 |
534 | 10.67.147.188:60456 |
535 | 10.67.180.153:34400 |
536 | 10.67.180.43:55852 |
537 | 10.67.138.231:57866 |
538 | 10.67.142.12:36966 |
539 | 10.67.137.31:59310 |
540 | 10.67.139.227:46276 |
541 | 10.67.151.182:43852 |
542 | 10.67.190.232:32976 |
543 | 10.67.131.88:42398 |
544 | 10.67.148.213:58268 |
545 | 10.67.172.239:43538 |
546 | 10.67.130.59:41304 |
547 | 10.67.168.14:43666 |
548 | 10.67.168.14:43674 |
549 | 10.67.132.33:43516 |
550 | 10.67.142.64:39290 |
551 | 10.67.182.175:47560 |
552 | 10.67.157.227:52836 |
553 | 10.67.188.82:37228 |
554 | 10.67.151.50:50004 |
555 | 10.67.170.24:60178 |
556 | 10.67.187.4:58316 |
557 | 10.67.136.68:41054 |
558 | 10.67.139.119:42248 |
559 | 10.67.181.184:34668 |
560 | 10.67.180.22:51320 |
561 | 10.67.145.172:54042 |
562 | 10.67.157.205:57002 |
563 | 10.67.154.25:33810 |
564 | 10.67.142.76:50954 |
565 | 10.67.144.41:51880 |
566 | 10.67.139.119:42254 |
567 | 10.67.150.2:43026 |
568 | 10.67.187.4:58322 |
569 | 10.67.131.88:44654 |
570 | 10.67.160.145:41340 |
571 | 10.67.187.4:58336 |
572 | 10.67.139.3:48316 |
573 | 10.67.137.254:47924 |
574 | 10.67.176.160:53854 |
575 | 10.67.153.18:40564 |
576 | 10.67.169.217:43470 |
577 | 10.67.189.251:53052 |
578 | 10.67.180.193:55200 |
579 | 10.67.129.164:39476 |
580 | 10.67.181.115:58804 |
581 | 10.67.162.3:56874 |
582 | 10.67.144.208:41784 |
583 | 10.67.157.24:49084 |
584 | 10.67.134.139:47116 |
585 | 10.67.139.47:51808 |
586 | 10.67.162.211:37312 |
587 | 10.67.171.202:37078 |
588 | 10.67.149.140:41182 |
589 | 10.67.160.43:37998 |
590 | 10.67.148.26:46990 |
591 | 10.67.180.40:42062 |
592 | 10.67.174.124:33602 |
593 | 10.67.168.14:32832 |
594 | 10.67.132.245:60188 |
595 | 10.67.132.4:56984 |
596 | 10.67.185.98:49532 |
597 | 10.67.142.17:37406 |
598 | 10.67.174.123:56474 |
599 | 10.67.151.50:54560 |
600 | 10.67.177.154:55704 |
601 | 10.67.135.125:40856 |
602 | 10.67.178.172:56350 |
603 | 10.67.174.189:45606 |
604 | 10.67.131.124:42612 |
605 | 10.67.188.63:41162 |
606 | 10.67.167.63:53278 |
607 | 10.67.163.70:44094 |
608 | 10.67.170.150:58844 |
609 | 10.67.140.139:59366 |
610 | 10.67.137.219:52626 |
611 | 10.67.181.138:40412 |
612 | 10.67.169.73:56428 |
613 | 10.67.157.48:44464 |
614 | 10.67.158.44:50990 |
615 | 10.67.188.105:33260 |
616 | 10.67.150.2:57566 |
617 | 10.67.175.154:46104 |
618 | 10.67.133.194:45890 |
619 | 10.67.144.41:54496 |
620 | 10.67.141.54:43810 |
621 | 10.67.181.175:48260 |
622 | 10.67.163.138:52748 |
623 | 10.67.150.27:60658 |
624 | 10.194.155.138:49708 |
625 | 10.67.137.233:35842 |
626 | 10.67.136.53:57690 |
627 | 10.67.183.155:47624 |
628 | 10.67.181.221:59224 |
629 | 10.67.138.48:40014 |
630 | 10.67.187.249:57210 |
631 | 10.67.137.9:60894 |
632 | 10.67.146.144:49096 |
633 | 10.67.162.3:40014 |
634 | 10.67.145.21:37242 |
635 | 10.67.185.162:43744 |
636 | 10.67.131.73:43126 |
637 | 10.67.149.130:52152 |
638 | 10.67.174.60:46080 |
639 | 10.67.142.179:58784 |
640 | 10.67.159.209:60366 |
641 | 10.67.132.4:43366 |
642 | 10.67.135.32:56382 |
643 | 10.67.159.90:41784 |
644 | 10.67.172.239:35610 |
645 | 10.67.134.162:49954 |
646 | 10.67.174.60:46082 |
647 | 10.67.134.162:49956 |
648 | 10.67.136.212:51926 |
649 | 10.67.135.183:50208 |
650 | 10.67.174.167:49526 |
651 | 10.67.153.18:36256 |
652 | 10.67.161.35:47020 |
653 | 10.67.149.218:47116 |
654 | 10.67.142.207:46182 |
655 | 10.67.144.231:49262 |
656 | 10.67.186.250:49326 |
657 | 10.67.136.71:48516 |
658 | 10.67.150.136:34248 |
659 | 10.67.148.229:60380 |
660 | 10.67.149.108:49868 |
661 | 10.67.146.71:34244 |
662 | 10.67.190.169:54276 |
663 | 10.67.150.67:46296 |
664 | 10.67.138.48:37668 |
665 | 10.67.134.159:59974 |
666 | 10.67.133.243:56620 |
667 | 10.67.183.198:40760 |
668 | 10.67.145.236:60744 |
669 | 10.67.174.77:41500 |
670 | 10.67.188.11:47680 |
671 | 10.67.132.33:55476 |
672 | 10.67.149.130:38692 |
673 | 10.67.130.8:56590 |
674 | 10.67.132.15:52064 |
675 | 10.67.130.21:37278 |
676 | 10.67.157.205:51050 |
677 | 10.67.190.123:41160 |
678 | 10.67.158.195:52318 |
679 | 10.67.180.134:60032 |
680 | 10.67.162.162:37478 |
681 | 10.67.149.108:49882 |
682 | 10.67.158.44:52426 |
683 | 10.67.137.254:56172 |
684 | 10.67.152.124:42100 |
685 | 10.67.177.130:49744 |
686 | 10.67.180.193:37404 |
687 | 10.67.177.141:58760 |
688 | 10.67.180.193:37410 |
689 | 10.67.184.0:43002 |
690 | 10.67.142.76:49280 |
691 | 10.67.132.121:39070 |
692 | 10.67.151.209:39586 |
693 | 10.67.129.105:48442 |
694 | 10.67.174.242:37196 |
695 | 10.67.187.138:42604 |
696 | 10.67.131.88:41716 |
697 | 10.67.135.182:57210 |
698 | 10.67.190.156:46696 |
699 | 10.67.147.188:52008 |
700 | 10.67.151.50:50674 |
701 | 10.67.152.148:35984 |
702 | 10.67.152.41:56724 |
703 | 10.67.149.150:34470 |
704 | 10.67.185.223:46784 |
705 | 10.67.164.18:53258 |
706 | 10.67.132.33:39288 |
707 | 10.67.134.139:51864 |
708 | 10.67.182.175:36662 |
709 | 10.67.174.167:58868 |
710 | 10.67.130.151:42620 |
711 | 10.67.153.223:46114 |
712 | 10.67.180.198:53282 |
713 | 10.67.176.85:54142 |
714 | 10.67.185.122:57892 |
715 | 10.67.181.115:49052 |
716 | 10.67.165.218:37772 |
717 | 10.67.191.249:47462 |
718 | 10.67.175.40:39004 |
719 | 10.67.138.42:52884 |
720 | 10.67.191.119:51930 |
721 | 10.67.133.194:54230 |
722 | 10.67.154.90:60438 |
723 | 10.67.174.176:42000 |
724 | 10.67.152.41:56428 |
725 | 10.67.175.177:56356 |
726 | 10.67.166.189:35446 |
727 | 10.67.131.120:57776 |
728 | 10.67.160.43:35044 |
729 | 10.67.174.242:52514 |
730 | 10.67.174.242:52512 |
731 | 10.67.130.149:59762 |
732 | 10.67.164.46:45452 |
733 | 10.67.168.118:55272 |
734 | 10.67.180.203:55538 |
735 | 10.67.180.198:52372 |
736 | 10.67.142.230:47700 |
737 | 10.67.162.211:40722 |
738 | 10.67.141.193:54888 |
739 | 10.67.146.244:38540 |
740 | 10.67.145.35:46312 |
741 | 10.67.157.86:47156 |
742 | 10.67.156.202:35414 |
743 | 10.67.178.147:36824 |
744 | 10.67.186.205:49646 |
745 | 10.67.180.40:60096 |
746 | 10.67.129.164:43936 |
747 | 10.67.145.255:48656 |
748 | 10.67.129.128:50964 |
749 | 10.67.139.70:56006 |
750 | 10.67.172.30:43024 |
751 | 10.67.163.172:50854 |
752 | 10.67.150.150:47408 |
753 | 10.67.175.177:46960 |
754 | 10.67.138.66:55046 |
755 | 10.67.154.48:53384 |
756 | 10.67.138.48:34828 |
757 | 10.67.176.164:49546 |
758 | 10.67.188.25:50222 |
759 | 10.67.175.154:43982 |
760 | 10.67.146.52:55266 |
761 | 10.67.153.193:34306 |
762 | 10.67.153.223:35898 |
763 | 10.67.141.235:40690 |
764 | 10.67.172.167:38014 |
765 | 10.67.169.216:59970 |
766 | 10.67.151.50:54294 |
767 | 10.67.183.192:45948 |
768 | 10.67.148.45:43906 |
769 | 10.67.132.116:46252 |
770 | 10.67.160.158:41736 |
771 | 10.67.141.193:50466 |
772 | 10.67.132.4:53774 |
773 | 10.67.166.97:40332 |
774 | 10.67.187.249:59218 |
775 | 10.67.188.73:53066 |
776 | 10.67.139.34:60596 |
777 | 10.67.190.37:43422 |
778 | 10.67.132.245:48898 |
779 | 10.67.135.125:40516 |
780 | 10.67.152.124:35324 |
781 | 10.67.130.21:33734 |
782 | 10.67.183.82:54208 |
783 | 10.67.137.143:35988 |
784 | 10.67.151.53:48752 |
785 | 10.67.140.185:33552 |
786 | 10.67.165.218:50464 |
787 | 10.67.152.148:33474 |
788 | 10.67.181.115:60798 |
789 | 10.67.190.204:37316 |
790 | 10.67.183.70:36960 |
791 | 10.67.128.252:40358 |
792 | 10.67.135.157:51954 |
793 | 10.67.145.255:35438 |
794 | 10.67.171.202:55636 |
795 | 10.67.183.97:33480 |
796 | 10.67.151.166:45846 |
797 | 10.67.187.26:53988 |
798 | 10.67.137.253:52818 |
799 | 10.67.183.147:42096 |
800 | 10.67.181.0:57126 |
801 | 10.67.134.74:50222 |
802 | 10.67.156.8:40192 |
803 | 10.67.136.112:49396 |
804 | 10.67.170.60:42116 |
805 | 10.67.133.158:52354 |
806 | 10.67.164.127:42812 |
807 | 10.67.183.70:36962 |
808 | 10.67.184.176:33450 |
809 | 10.67.166.189:47232 |
810 | 10.67.152.219:50202 |
811 | 10.67.133.194:50818 |
812 | 10.67.175.40:46780 |
813 | 10.67.157.112:34652 |
814 | 10.67.160.145:34404 |
815 | 10.67.160.158:38720 |
816 | 10.67.134.74:50232 |
817 | 10.67.143.225:33222 |
818 | 10.67.166.189:47236 |
819 | 10.67.141.226:49902 |
820 | 10.67.167.78:57972 |
821 | 10.67.144.135:32956 |
822 | 10.67.190.123:47164 |
823 | 10.67.133.158:52356 |
824 | 10.67.153.115:60342 |
825 | 10.67.146.159:33148 |
826 | 10.67.183.218:33728 |
827 | 10.67.163.168:58012 |
828 | 10.67.154.85:50220 |
829 | 10.67.160.53:55394 |
830 | 10.67.157.86:39802 |
831 | 10.67.139.61:36598 |
832 | 10.67.138.251:45232 |
833 | 10.67.176.115:42540 |
834 | 10.67.174.233:47538 |
835 | 10.67.172.239:54844 |
836 | 10.67.147.135:52068 |
837 | 10.67.184.9:60022 |
838 | 10.67.172.30:52794 |
839 | 10.67.187.4:33188 |
840 | 10.67.184.176:40852 |
841 | 10.67.150.67:37634 |
842 | 10.67.128.207:50788 |
843 | 10.67.161.35:41258 |
844 | 10.67.181.195:34182 |
845 | 10.67.161.221:43920 |
846 | 10.67.152.241:39618 |
847 | 10.67.174.233:47554 |
848 | 10.67.133.109:44716 |
849 | 10.67.169.73:60124 |
850 | 10.67.169.216:51938 |
851 | 10.67.161.221:44256 |
852 | 10.67.150.252:54002 |
853 | 10.67.177.141:47814 |
854 | 10.67.162.185:35766 |
855 | 10.67.172.167:55694 |
856 | 10.67.172.214:45342 |
857 | 10.67.142.236:54966 |
858 | 10.67.139.61:46830 |
859 | 10.67.170.188:33170 |
860 | 10.67.168.171:52084 |
861 | 10.67.184.143:58942 |
862 | 10.67.130.8:50252 |
863 | 10.67.154.220:35456 |
864 | 10.67.143.147:56028 |
865 | 10.67.146.254:42150 |
866 | 10.67.144.147:56546 |
867 | 10.67.146.207:52940 |
868 | 10.67.176.36:57298 |
869 | 10.67.165.205:43018 |
870 | 10.67.134.12:37576 |
871 | 10.194.131.107:35290 |
872 | 10.67.146.159:36566 |
873 | 10.67.138.110:54360 |
874 | 10.67.170.15:52326 |
875 | 10.67.150.34:34432 |
876 | 10.67.157.227:42344 |
877 | 10.67.181.133:40798 |
878 | 10.67.136.112:59250 |
879 | 10.67.154.60:45732 |
880 | 10.67.161.53:52248 |
881 | 10.67.150.15:59818 |
882 | 10.67.161.35:57334 |
883 | 10.67.132.33:38056 |
884 | 10.67.184.143:58958 |
885 | 10.67.134.12:37584 |
886 | 10.67.174.173:51382 |
887 | 10.67.144.135:60248 |
888 | 10.67.145.172:35510 |
889 | 10.67.152.133:35604 |
890 | 10.67.166.153:39696 |
891 | 10.67.156.8:47982 |
892 | 10.67.143.217:54870 |
893 | 10.67.142.69:57896 |
894 | 10.67.148.152:37868 |
895 | 10.67.136.71:44786 |
896 | 10.67.187.12:51264 |
897 | 10.67.185.98:56128 |
898 | 10.67.186.205:43918 |
899 | 10.67.171.223:38716 |
900 | 10.67.148.220:54556 |
901 | 10.67.139.47:34116 |
902 | 10.67.146.89:52886 |
903 | 10.67.167.63:43424 |
904 | 10.67.138.109:51020 |
905 | 10.67.184.231:60516 |
906 | 10.67.168.79:57988 |
907 | 10.67.190.232:40498 |
908 | 10.67.152.204:60616 |
909 | 10.67.181.241:39638 |
910 | 10.67.185.230:53486 |
911 | 10.67.178.183:60120 |
912 | 10.67.135.183:55162 |
913 | 10.67.188.3:53256 |
914 | 10.67.177.154:59546 |
915 | 10.67.154.150:54414 |
916 | 10.67.182.175:58574 |
917 | 10.67.157.86:37598 |
918 | 10.67.141.43:33820 |
919 | 10.67.172.214:34428 |
920 | 10.67.167.115:57292 |
921 | 10.67.147.132:50152 |
922 | 10.67.186.233:59844 |
923 | 10.67.187.150:41646 |
924 | 10.67.154.28:58076 |
925 | 10.67.152.124:46812 |
926 | 10.67.151.156:60228 |
927 | 10.67.142.236:49126 |
928 | 10.67.138.217:53316 |
929 | 10.67.186.131:51846 |
930 | 10.67.148.249:37618 |
931 | 10.67.134.64:55778 |
932 | 10.67.174.119:51700 |
933 | 10.67.185.93:51204 |
934 | 10.67.154.220:34968 |
935 | 10.67.145.213:45966 |
936 | 10.67.129.137:42864 |
937 | 10.67.141.50:57906 |
938 | 10.67.141.93:46108 |
939 | 10.67.169.105:58600 |
940 | 10.67.150.150:60840 |
941 | 10.67.135.183:33120 |
942 | 10.67.178.172:35632 |
943 | 10.67.162.211:51522 |
944 | 10.67.164.18:42214 |
945 | 10.67.176.36:38116 |
946 | 10.67.153.223:37032 |
947 | 10.67.146.75:59152 |
948 | 10.67.172.214:47230 |
949 | 10.67.177.224:55174 |
950 | 10.67.153.118:46414 |
951 | 10.67.181.0:53548 |
952 | 10.67.131.250:57922 |
953 | 10.67.149.218:44842 |
954 | 10.67.139.175:51840 |
955 | 10.67.177.224:55182 |
956 | 10.67.160.151:38688 |
957 | 10.67.154.60:59506 |
958 | 10.67.149.130:33726 |
959 | 10.67.188.249:56198 |
960 | 10.67.189.208:49050 |
961 | 10.67.134.23:35896 |
962 | 10.67.136.135:44150 |
963 | 10.67.151.53:34344 |
964 | 10.67.188.11:54508 |
965 | 10.67.189.232:47972 |
966 | 10.67.148.229:38526 |
967 | 10.67.140.215:51532 |
968 | 10.67.143.217:34278 |
969 | 10.67.142.124:55184 |
970 | 10.67.158.23:54908 |
971 | 10.67.142.69:51054 |
972 | 10.67.191.249:43390 |
973 | 10.194.182.1:58148 |
974 | 10.67.183.155:34594 |
975 | 10.67.143.197:54294 |
976 | 10.67.130.204:58212 |
977 | 10.67.146.144:60036 |
978 | 10.67.156.8:41874 |
979 | 10.67.132.33:41004 |
980 | 10.67.172.168:50968 |
981 | 10.67.167.78:50352 |
982 | 10.67.172.168:50954 |
983 | 10.67.164.144:38412 |
984 | 10.67.136.190:53888 |
985 | 10.67.152.171:54406 |
986 | 10.67.136.73:39030 |
987 | 10.67.130.21:56668 |
988 | 10.67.138.231:37350 |
989 | 10.67.144.215:48914 |
990 | 10.67.188.105:46468 |
991 | 10.67.134.139:57852 |
992 | 10.67.158.225:54038 |
993 | 10.67.130.151:54184 |
994 | 10.67.136.183:56920 |
995 | 10.67.186.184:37804 |
996 | 10.67.174.0:56550 |
997 | 10.67.174.60:49720 |
998 | 10.67.146.207:34896 |
999 | 10.67.156.52:54250 |
1000 | 10.67.150.34:51688 |
1001 | 10.67.168.40:55920 |
1002 | 10.67.152.171:54408 |
1003 | 10.67.188.43:60154 |
1004 | 10.67.168.14:55846 |
1005 | 10.67.145.222:41488 |
1006 | 10.67.189.208:46794 |
1007 | 10.67.153.115:50846 |
1008 | 10.67.154.220:53984 |
1009 | 10.67.183.218:40886 |
1010 | 10.67.176.16:56252 |
1011 | 10.67.152.124:40136 |
1012 | 10.67.148.15:34090 |
1013 | 10.67.178.113:40722 |
1014 | 10.67.177.229:48910 |
1015 | 10.67.142.162:42582 |
1016 | 10.67.159.44:45776 |
1017 | 10.67.138.195:36158 |
1018 | 10.67.145.21:45514 |
1019 | 10.67.142.64:55972 |
1020 | 10.67.183.198:53584 |
1021 | 10.67.142.124:45626 |
1022 | 10.67.140.87:47742 |
1023 | 10.67.176.115:41330 |
1024 | 10.67.131.120:56316 |
1025 | 10.67.180.161:58086 |
1026 | 10.67.148.220:46054 |
1027 | 10.67.176.115:41336 |
1028 | 10.67.180.224:44356 |
1029 | 10.67.138.48:57062 |
1030 | 10.67.132.214:41478 |
1031 | 10.67.162.185:44220 |
1032 | 10.67.149.218:47572 |
1033 | 10.67.183.97:48244 |
1034 | 10.67.152.148:49124 |
1035 | 10.67.137.94:37448 |
1036 | 10.67.190.204:55450 |
1037 | 10.67.190.232:42998 |
1038 | 10.67.136.182:50506 |
1039 | 10.67.186.131:42060 |
1040 | 10.67.140.139:44912 |
1041 | 10.67.150.136:43130 |
1042 | 10.67.158.47:55548 |
1043 | 10.67.190.98:51614 |
1044 | 10.67.175.40:51722 |
1045 | 10.194.162.150:56456 |
1046 | 10.67.133.191:53646 |
1047 | 10.67.150.136:56792 |
1048 | 10.67.133.93:46972 |
1049 | 10.67.181.241:36702 |
1050 | 10.67.134.64:51202 |
1051 | 10.67.137.253:47946 |
1052 | 10.67.129.103:39980 |
1053 | 10.67.164.83:49638 |
1054 | 10.67.153.207:58956 |
1055 | 10.67.135.125:51436 |
1056 | 10.67.188.112:53332 |
1057 | 10.67.174.173:50646 |
1058 | 10.67.176.115:39206 |
1059 | 10.67.133.109:50946 |
1060 | 10.67.145.71:35958 |
1061 | 10.67.172.167:57448 |
1062 | 10.67.174.119:35528 |
1063 | 10.67.132.214:51926 |
1064 | 10.67.181.92:45856 |
1065 | 10.67.143.217:41834 |
1066 | 10.67.150.27:46226 |
1067 | 10.67.140.112:59484 |
1068 | 10.67.145.44:40112 |
1069 | 10.67.148.216:39086 |
1070 | 10.67.184.55:47702 |
1071 | 10.67.129.137:49268 |
1072 | 10.67.190.38:48766 |
1073 | 10.67.148.53:55806 |
1074 | 10.67.190.96:34564 |
1075 | 10.67.144.49:42484 |
1076 | 10.67.180.224:34736 |
1077 | 10.67.139.13:46794 |
1078 | 10.67.184.55:57454 |
1079 | 10.67.135.122:43548 |
1080 | 10.67.133.243:34288 |
1081 | 10.67.145.21:60312 |
1082 | 10.67.165.218:35528 |
1083 | 10.67.156.8:33258 |
1084 | 10.67.162.191:52854 |
1085 | 10.67.172.168:55064 |
1086 | 10.67.181.96:53948 |
1087 | 10.67.143.217:41840 |
1088 | 10.67.148.50:38836 |
1089 | 10.67.157.35:35116 |
1090 | 10.67.181.187:39718 |
1091 | 10.67.154.17:47744 |
1092 | 10.67.168.40:53904 |
1093 | 10.67.132.238:54718 |
1094 | 10.67.183.147:41186 |
1095 | 10.67.143.217:46330 |
1096 | 10.194.175.16:56048 |
1097 | 10.67.148.233:51144 |
1098 | 10.67.159.75:34358 |
1099 | 10.67.140.112:48040 |
1100 | 10.67.183.192:35924 |
1101 | 10.67.145.255:57086 |
1102 | 10.67.167.12:57772 |
1103 | 10.67.188.105:43778 |
1104 | 10.67.136.190:45430 |
1105 | 10.67.134.85:57112 |
1106 | 10.67.164.18:52750 |
1107 | 10.67.136.53:32998 |
1108 | 10.67.133.93:48118 |
1109 | 10.67.159.75:52904 |
1110 | 10.67.191.126:43912 |
1111 | 10.67.148.216:35754 |
1112 | 10.67.151.17:45290 |
1113 | 10.67.145.255:56756 |
1114 | 10.67.128.207:47746 |
1115 | 10.67.167.12:46524 |
1116 | 10.67.129.105:48940 |
1117 | 10.67.158.21:33250 |
1118 | 10.67.181.105:44860 |
1119 | 10.67.190.175:39242 |
1120 | 10.67.188.239:39986 |
1121 | 10.67.149.150:41760 |
1122 | 10.67.187.12:47686 |
1123 | 10.67.145.174:53174 |
1124 | 10.67.181.92:34704 |
1125 | 10.67.154.28:37132 |
1126 | 10.67.162.206:38794 |
1127 | 10.67.162.223:35604 |
1128 | 10.67.176.164:59336 |
1129 | 10.67.154.30:47128 |
1130 | 10.67.163.236:57932 |
1131 | 10.67.130.241:33006 |
1132 | 10.67.185.214:48718 |
1133 | 10.67.174.119:56384 |
1134 | 10.67.137.233:57882 |
1135 | 10.67.143.197:40598 |
1136 | 10.67.161.214:50032 |
1137 | 10.67.152.143:49296 |
1138 | 10.67.149.98:46336 |
1139 | 10.67.184.231:56010 |
1140 | 10.67.188.3:41590 |
1141 | 10.67.170.60:34096 |
1142 | 10.67.188.11:57252 |
1143 | 10.67.174.189:59634 |
1144 | 10.67.174.167:55392 |
1145 | 10.67.172.214:51790 |
1146 | 10.67.136.135:33350 |
1147 | 10.67.168.72:46438 |
1148 | 10.67.163.249:37906 |
1149 | 10.67.190.221:49428 |
1150 | 10.67.163.79:44900 |
1151 | 10.67.134.139:59818 |
1152 | 10.67.158.38:34924 |
1153 | 10.67.157.205:38834 |
1154 | 10.67.154.25:41118 |
1155 | 10.67.132.121:50814 |
1156 | 10.67.136.112:56956 |
1157 | 10.67.131.11:46028 |
1158 | 10.67.187.139:48772 |
1159 | 10.67.160.145:44522 |
1160 | 10.67.139.127:34920 |
1161 | 10.67.141.100:41076 |
1162 | 10.67.136.217:55908 |
1163 | 10.67.163.223:45398 |
1164 | 10.67.159.75:45788 |
1165 | 10.67.150.136:38692 |
1166 | 10.67.144.49:47170 |
1167 | 10.67.146.71:49610 |
1168 | 10.67.131.120:42100 |
1169 | 10.67.159.25:40778 |
1170 | 10.67.148.45:44132 |
1171 | 10.67.169.93:43116 |
1172 | 10.67.142.143:43946 |
1173 | 10.67.136.73:48986 |
1174 | 10.67.132.116:55420 |
1175 | 10.67.135.181:35850 |
1176 | 10.67.185.78:45086 |
1177 | 10.67.169.93:59916 |
1178 | 10.67.174.154:55936 |
1179 | 10.67.161.30:37366 |
1180 | 10.67.136.73:39494 |
1181 | 10.67.181.96:55696 |
1182 | 10.67.161.30:54166 |
1183 | 10.67.149.98:50814 |
1184 | 10.67.185.219:52022 |
1185 | 10.67.165.205:55512 |
1186 | 10.67.173.239:36882 |
1187 | 10.67.170.172:35542 |
1188 | 10.67.181.201:45652 |
1189 | 10.67.176.160:41272 |
1190 | 10.67.159.40:60206 |
1191 | 10.67.144.175:45494 |
1192 | 10.67.176.99:50370 |
1193 | 10.67.191.119:45192 |
1194 | 10.67.166.101:42928 |
1195 | 10.67.135.125:58194 |
1196 | 10.67.128.155:54144 |
1197 | 10.67.150.2:59454 |
1198 | 10.67.190.34:32930 |
1199 | 10.67.145.236:49356 |
1200 | 10.67.140.77:53154 |
1201 | 10.67.158.76:59448 |
1202 | 10.67.167.12:47174 |
1203 | 10.67.132.214:53320 |
1204 | 10.67.139.119:44558 |
1205 | 10.67.131.76:39106 |
1206 | 10.67.170.172:49710 |
1207 | 10.67.160.249:46812 |
1208 | 10.67.180.43:44914 |
1209 | 10.67.174.77:34590 |
1210 | 10.67.145.198:35226 |
1211 | 10.67.137.94:57958 |
1212 | 10.67.182.154:47988 |
1213 | 10.67.179.110:55580 |
1214 | 10.67.149.150:40780 |
1215 | 10.67.149.98:37098 |
1216 | 10.67.145.236:48760 |
1217 | 10.67.153.196:48228 |
1218 | 10.67.137.180:58992 |
1219 | 10.67.161.137:39564 |
1220 | 10.67.147.163:35158 |
1221 | 10.67.149.71:48388 |
1222 | 10.67.153.58:39506 |
1223 | 10.67.161.127:43600 |
1224 | 10.67.146.71:43412 |
1225 | 10.67.174.154:48480 |
1226 | 10.67.156.49:52612 |
1227 | 10.67.151.3:52860 |
1228 | 10.67.163.72:40880 |
1229 | 10.67.170.24:58412 |
1230 | 10.67.161.116:56204 |
1231 | 10.67.188.136:50886 |
1232 | 10.67.149.108:53422 |
1233 | 10.67.167.72:42846 |
1234 | 10.67.181.175:33228 |
1235 | 10.67.132.116:60522 |
1236 | 10.67.134.46:57076 |
1237 | 10.67.144.226:47758 |
1238 | 10.67.151.3:57528 |
Quick update from another occurrence starting at ~ 20:45 UTC today:
The processlist polling that @CDanis set up worked quite nicely here. Specifically, if you look at the sample files in /home/cdanis/processlist and do something simple like count threads not in Command: Sleep, you see something like:
2024-08-14T20:45:30+00:00 20 2024-08-14T20:45:45+00:00 237 2024-08-14T20:46:00+00:00 489 2024-08-14T20:46:15+00:00 692 2024-08-14T20:46:30+00:00 764 2024-08-14T20:46:45+00:00 833 2024-08-14T20:47:00+00:00 882 2024-08-14T20:47:15+00:00 965 2024-08-14T20:47:31+00:00 1042 2024-08-14T20:47:46+00:00 1084 2024-08-14T20:48:01+00:00 1156 2024-08-14T20:48:16+00:00 1255 2024-08-14T20:48:31+00:00 1482 2024-08-14T20:48:46+00:00 1543 2024-08-14T20:49:01+00:00 1583 2024-08-14T20:49:17+00:00 1623 2024-08-14T20:49:32+00:00 1663 2024-08-14T20:49:47+00:00 1706 2024-08-14T20:50:02+00:00 1744 2024-08-14T20:50:18+00:00 1789 2024-08-14T20:50:33+00:00 1938 2024-08-14T20:55:39+00:00 1939 2024-08-14T21:00:00+00:00 548 2024-08-14T21:04:36+00:00 66 2024-08-14T21:04:51+00:00 26
Setting aside the possibility of missing a short-lived precursor event that cleared before we could see it, the 20:45:45 sample looks promising.
Looking at that processlist, of the 219 commonswiki queries present, 167 are UserEditCountUpdate::doUpdate for the same exact user_id. Of the remaining 52 queries, 40 are ChangeTagsStore::updateTags predominantly related to Cat-a-lot, which we believe this user was using at the time.
These eventually clear out by the 20:55:39 sample, which seems to be a bit before db1160 appears to fully recover some time around / after 21:00. These queries also seem to be rolling back quite slowly (e.g., a given query might be in rollback for 15-30s, with only 1 or 2 in Rollback per sample over the majority of the bad period).
Of the 12 remaining queries present in the 20:45:45 sample, I've spot-checked what happens to them in later samples (e.g., by tracking the same client ID), and they all appear to clear out quickly - either with no subsequent queries on that ID, or replaced with a UserEditCountUpdate::doUpdate shortly after.
Anyway, while far from a definitive culprit, these queries do seem to be the very first to get "stuck" after which further queries quickly pile up. It's unclear to me why these in particular would be special, as they seem to be rather simple in practice.
Paste containing the 20:45:45 sample: https://phabricator.wikimedia.org/P67322
This is most likely unrelated, Thumbor doesn't really touch the databases
This happened again, but it was very brief and didn't generate any pages.
Things I saw:
- Hundreds of DELETEs to linter like this one:
| 3985847190 | wikiuser2023 | 10.67.176.184:39952 | commonswiki | Query | 166 | Updating | DELETE /* MediaWiki\Linter\Database::setForPage */ FROM `linter` WHERE linter_page = 145347895 | 0.000 |
- Hundreds of SELECTs again on the master, to the page table similar to this
| 3985845073 | wikiuser2023 | 10.67.137.19:45296 | commonswiki | Query | 261 | Statistics | SELECT /* WikiPage::lockAndGetLatest */ page_latest FROM `page` WHERE page_id = 13828077 AND p | 0.000 |
List of PODS generating writes:
1 | 10.64.16.77:48882 |
---|---|
2 | 10.67.132.180:56876 |
3 | 10.67.147.29:60142 |
4 | 10.67.152.212:51008 |
5 | 10.67.149.144:41394 |
6 | 10.67.133.174:52418 |
7 | 10.67.135.118:43748 |
8 | 10.67.191.124:49726 |
9 | 10.67.129.10:55766 |
10 | 10.67.176.22:43972 |
11 | 10.67.136.14:40968 |
12 | 10.67.181.179:57398 |
13 | 10.67.151.1:58222 |
14 | 10.67.184.170:60616 |
15 | 10.67.139.85:50278 |
16 | 10.67.137.236:35996 |
17 | 10.67.188.79:55782 |
18 | 10.67.130.142:52586 |
19 | 10.67.163.97:41066 |
20 | 10.67.161.118:46818 |
21 | 10.67.148.220:48084 |
22 | 10.67.158.244:35176 |
23 | 10.67.149.12:54384 |
24 | 10.67.163.188:51048 |
25 | 10.67.135.118:43758 |
26 | 10.67.144.201:57818 |
27 | 10.67.163.233:43212 |
28 | 10.67.163.113:40090 |
29 | 10.67.186.167:43710 |
30 | 10.67.166.24:45796 |
31 | 10.67.179.93:53598 |
32 | 10.67.149.207:44920 |
33 | 10.67.185.142:37822 |
34 | 10.67.168.146:37928 |
35 | 10.67.168.146:37942 |
36 | 10.67.144.103:36588 |
37 | 10.67.129.72:52132 |
38 | 10.67.164.5:57680 |
39 | 10.67.137.236:36012 |
40 | 10.67.145.104:43610 |
41 | 10.67.152.212:51016 |
42 | 10.67.152.212:51018 |
43 | 10.67.148.76:49600 |
44 | 10.67.184.33:38754 |
45 | 10.67.133.56:49014 |
46 | 10.67.185.142:37830 |
47 | 10.67.158.29:42236 |
48 | 10.67.144.189:55212 |
49 | 10.67.154.9:39064 |
50 | 10.67.157.76:39022 |
51 | 10.67.140.244:57876 |
52 | 10.67.163.113:40100 |
53 | 10.67.145.104:43614 |
54 | 10.67.148.147:52974 |
55 | 10.67.181.118:60000 |
56 | 10.67.184.170:60632 |
57 | 10.67.161.118:46832 |
58 | 10.67.133.174:52434 |
59 | 10.67.153.156:55052 |
60 | 10.67.158.244:35182 |
61 | 10.67.179.208:46028 |
62 | 10.67.159.48:47150 |
63 | 10.67.176.22:43986 |
64 | 10.67.188.79:55798 |
65 | 10.67.180.222:41476 |
66 | 10.67.152.141:40170 |
67 | 10.67.158.244:35190 |
68 | 10.67.133.244:58790 |
69 | 10.67.162.240:49492 |
70 | 10.67.177.227:48620 |
71 | 10.67.129.5:42024 |
72 | 10.67.159.183:45718 |
73 | 10.67.135.118:43768 |
74 | 10.67.129.162:40490 |
75 | 10.67.136.14:40970 |
76 | 10.67.188.79:55812 |
77 | 10.67.184.170:60640 |
78 | 10.67.156.167:43570 |
79 | 10.67.163.97:41070 |
80 | 10.67.180.3:60894 |
81 | 10.67.185.96:33902 |
82 | 10.67.137.236:51770 |
83 | 10.67.132.48:36146 |
84 | 10.67.147.150:37666 |
85 | 10.67.179.208:46044 |
86 | 10.67.161.118:46838 |
87 | 10.67.145.62:39986 |
88 | 10.67.172.250:50986 |
89 | 10.67.180.222:41484 |
90 | 10.67.138.131:40270 |
91 | 10.67.139.85:50288 |
92 | 10.67.162.47:45926 |
93 | 10.67.162.47:45930 |
94 | 10.67.158.29:42244 |
95 | 10.67.161.77:41652 |
96 | 10.67.149.12:37252 |
97 | 10.67.130.220:60998 |
98 | 10.67.138.40:33150 |
99 | 10.67.143.20:53424 |
100 | 10.67.185.206:57392 |
101 | 10.67.129.5:36686 |
102 | 10.67.181.118:60962 |
103 | 10.67.148.86:47400 |
104 | 10.67.160.148:48290 |
105 | 10.67.175.43:39476 |
106 | 10.67.171.202:49878 |
107 | 10.67.143.20:53430 |
108 | 10.67.144.147:40976 |
109 | 10.67.172.6:33358 |
110 | 10.67.180.228:47954 |
111 | 10.67.131.235:35914 |
112 | 10.67.162.46:45586 |
113 | 10.67.152.191:37408 |
114 | 10.67.148.180:46940 |
115 | 10.67.137.30:47826 |
116 | 10.67.150.242:50634 |
117 | 10.67.183.100:32820 |
118 | 10.67.157.35:41422 |
119 | 10.67.174.25:36248 |
120 | 10.67.166.226:34660 |
121 | 10.67.161.232:56098 |
122 | 10.67.136.54:44954 |
123 | 10.67.132.176:60474 |
124 | 10.67.186.167:41184 |
125 | 10.67.160.38:36668 |
126 | 10.67.158.66:57850 |
127 | 10.67.170.178:56538 |
128 | 10.67.188.111:38328 |
129 | 10.67.190.78:40906 |
130 | 10.67.190.78:40910 |
131 | 10.67.152.191:37412 |
132 | 10.67.191.235:44586 |
133 | 10.67.140.147:58450 |
134 | 10.67.157.20:53666 |
135 | 10.67.138.1:40532 |
136 | 10.67.151.6:45174 |
137 | 10.67.147.150:40152 |
138 | 10.67.146.93:40224 |
139 | 10.67.183.100:33594 |
140 | 10.67.162.170:45852 |
141 | 10.67.142.110:38678 |
142 | 10.67.131.222:52732 |
143 | 10.67.140.194:57812 |
144 | 10.67.180.61:42208 |
145 | 10.67.152.191:37414 |
146 | 10.67.157.251:49758 |
147 | 10.67.133.223:50790 |
148 | 10.67.142.15:55850 |
149 | 10.67.139.143:60578 |
150 | 10.67.182.172:42924 |
151 | 10.67.132.176:60476 |
152 | 10.67.138.167:58346 |
153 | 10.67.157.96:58652 |
154 | 10.67.129.137:38536 |
155 | 10.67.162.235:41634 |
156 | 10.67.189.223:41054 |
157 | 10.67.175.43:39478 |
158 | 10.67.159.23:55374 |
159 | 10.67.133.180:49364 |
160 | 10.67.162.26:47544 |
161 | 10.67.168.3:46902 |
162 | 10.67.185.155:37462 |
163 | 10.67.140.77:43750 |
164 | 10.67.191.121:54984 |
165 | 10.67.172.185:40396 |
166 | 10.67.170.178:56546 |
167 | 10.67.148.147:38720 |
168 | 10.67.180.228:47982 |
169 | 10.67.170.142:52526 |
170 | 10.67.160.85:42914 |
171 | 10.67.159.183:56962 |
172 | 10.67.129.149:58148 |
173 | 10.67.135.236:50560 |
174 | 10.67.162.26:43600 |
175 | 10.67.170.178:56554 |
176 | 10.67.184.0:56096 |
177 | 10.67.164.73:52728 |
178 | 10.67.161.83:47610 |
179 | 10.67.171.30:43034 |
180 | 10.67.185.142:40846 |
181 | 10.67.190.140:50242 |
182 | 10.67.164.36:35078 |
183 | 10.67.174.128:40270 |
184 | 10.67.160.180:36684 |
185 | 10.67.169.254:33722 |
186 | 10.67.179.223:56762 |
187 | 10.67.177.134:50134 |
188 | 10.67.131.46:38996 |
189 | 10.67.171.223:44744 |
190 | 10.67.132.245:44460 |
191 | 10.67.149.180:40122 |
192 | 10.67.143.20:47118 |
193 | 10.67.161.232:56104 |
194 | 10.67.138.9:38966 |
195 | 10.67.138.131:54620 |
196 | 10.67.162.46:36012 |
197 | 10.67.176.15:49062 |
198 | 10.67.153.156:43352 |
199 | 10.67.140.194:57064 |
200 | 10.67.148.192:60238 |
201 | 10.67.180.61:44316 |
202 | 10.67.170.142:44836 |
203 | 10.67.167.58:52700 |
204 | 10.67.148.50:54316 |
205 | 10.67.143.87:36550 |
206 | 10.67.162.170:41038 |
207 | 10.67.159.48:39812 |
208 | 10.67.191.68:52904 |
209 | 10.67.177.175:48388 |
210 | 10.67.135.8:32882 |
211 | 10.67.186.126:37322 |
212 | 10.67.162.235:44606 |
213 | 10.67.158.231:39114 |
214 | 10.67.168.105:55078 |
215 | 10.67.188.91:37462 |
216 | 10.67.162.46:55400 |
217 | 10.67.178.85:37268 |
218 | 10.67.160.118:41206 |
219 | 10.67.129.5:32952 |
220 | 10.67.140.187:42256 |
221 | 10.67.183.100:44394 |
222 | 10.67.138.211:54474 |
223 | 10.67.181.200:42936 |
224 | 10.67.140.194:42454 |
225 | 10.67.147.29:43020 |
226 | 10.67.164.179:44026 |
227 | 10.67.132.32:50282 |
228 | 10.67.181.104:42130 |
229 | 10.67.140.194:42470 |
230 | 10.67.146.189:49414 |
231 | 10.67.134.123:47494 |
232 | 10.67.139.50:50260 |
233 | 10.67.140.188:40338 |
234 | 10.64.0.155:35256 |
235 | 10.67.183.187:55352 |
236 | 10.67.164.131:38086 |
237 | 10.67.185.206:56936 |
238 | 10.67.160.85:43484 |
239 | 10.67.179.254:58142 |
240 | 10.67.157.35:53520 |
241 | 10.67.161.231:34390 |
242 | 10.67.188.111:52348 |
243 | 10.67.130.129:55952 |
244 | 10.67.181.131:43928 |
245 | 10.67.157.45:57962 |
246 | 10.67.176.86:46378 |
247 | 10.67.161.34:57404 |
248 | 10.67.143.193:46124 |
249 | 10.67.140.7:40408 |
250 | 10.67.143.193:46134 |
251 | 10.67.162.26:33556 |
252 | 10.67.129.149:41934 |
253 | 10.67.138.1:46356 |
254 | 10.67.164.39:41554 |
255 | 10.67.175.57:40848 |
256 | 10.67.133.180:53636 |
257 | 10.67.129.149:41946 |
258 | 10.67.140.194:38972 |
259 | 10.67.164.36:37836 |
260 | 10.67.162.183:39790 |
261 | 10.67.159.82:47522 |
262 | 10.67.187.244:41836 |
263 | 10.67.148.194:59056 |
264 | 10.67.181.200:54938 |
265 | 10.67.129.5:58368 |
266 | 10.67.160.56:57714 |
267 | 10.67.160.15:47300 |
268 | 10.67.149.64:37090 |
269 | 10.67.157.98:60604 |
270 | 10.67.172.243:46050 |
271 | 10.67.161.77:54420 |
272 | 10.67.167.15:37800 |
273 | 10.67.130.81:50502 |
274 | 10.67.176.165:47790 |
275 | 10.67.180.179:46566 |
276 | 10.67.143.222:42358 |
277 | 10.67.133.157:46112 |
278 | 10.67.149.98:45706 |
279 | 10.67.136.14:57866 |
280 | 10.67.163.188:58830 |
281 | 10.67.187.244:41842 |
282 | 10.67.185.142:56434 |
283 | 10.67.162.144:58458 |
284 | 10.67.168.146:40370 |
285 | 10.67.169.126:37360 |
286 | 10.67.140.194:56780 |
287 | 10.67.181.200:53826 |
288 | 10.67.188.43:40740 |
289 | 10.67.160.118:55074 |
290 | 10.67.146.89:45772 |
291 | 10.67.152.191:33268 |
292 | 10.67.144.137:33968 |
293 | 10.67.153.28:50932 |
294 | 10.67.164.73:51910 |
295 | 10.67.175.57:36804 |
296 | 10.67.145.208:44500 |
297 | 10.67.170.178:57782 |
298 | 10.67.179.254:38452 |
299 | 10.67.137.178:44504 |
300 | 10.67.170.178:57788 |
301 | 10.67.164.73:51914 |
302 | 10.67.180.254:49676 |
303 | 10.67.137.242:47674 |
304 | 10.67.160.38:37466 |
305 | 10.67.151.239:43738 |
306 | 10.67.160.56:56030 |
307 | 10.67.191.124:50380 |
308 | 10.67.129.149:40222 |
309 | 10.67.149.144:55360 |
310 | 10.67.139.255:34674 |
311 | 10.67.152.212:59064 |
312 | 10.67.168.3:44476 |
313 | 10.67.143.84:32802 |
314 | 10.67.144.124:52002 |
315 | 10.67.128.221:51424 |
316 | 10.67.161.118:60260 |
317 | 10.67.142.232:58602 |
318 | 10.67.164.36:60150 |
319 | 10.67.139.50:49276 |
320 | 10.67.138.1:38722 |
321 | 10.67.188.55:54714 |
322 | 10.67.143.193:47914 |
323 | 10.67.130.12:36160 |
324 | 10.67.149.144:59242 |
325 | 10.67.145.208:35170 |
326 | 10.67.142.15:53248 |
327 | 10.67.175.16:43954 |
328 | 10.67.186.203:33970 |
329 | 10.67.147.30:34866 |
330 | 10.67.135.96:38668 |
331 | 10.67.129.79:46612 |
332 | 10.67.189.201:55908 |
333 | 10.67.136.202:36630 |
334 | 10.67.138.209:44512 |
335 | 10.67.180.228:47840 |
336 | 10.67.171.46:45518 |
337 | 10.67.175.128:43152 |
338 | 10.67.188.189:41174 |
339 | 10.67.144.230:43680 |
340 | 10.67.184.25:55208 |
341 | 10.67.180.217:53682 |
342 | 10.67.190.23:33412 |
343 | 10.67.132.180:55148 |
344 | 10.67.142.232:56704 |
345 | 10.67.190.180:36690 |
346 | 10.67.180.221:50332 |
347 | 10.67.150.144:56272 |
348 | 10.67.134.106:53764 |
349 | 10.67.190.180:36706 |
350 | 10.67.156.137:43126 |
351 | 10.67.158.66:56876 |
352 | 10.67.143.20:33418 |
353 | 10.67.172.232:47750 |
354 | 10.67.135.75:58040 |
355 | 10.67.177.214:42970 |
356 | 10.67.140.147:40608 |
357 | 10.67.130.18:38624 |
358 | 10.67.149.9:41098 |
359 | 10.67.145.221:36108 |
360 | 10.67.183.47:36610 |
361 | 10.67.164.36:35906 |
362 | 10.67.175.86:54344 |
363 | 10.67.156.167:37468 |
364 | 10.67.191.68:45974 |
365 | 10.67.139.85:51332 |
366 | 10.67.162.144:44158 |
367 | 10.67.175.54:57676 |
368 | 10.67.190.143:47514 |
369 | 10.67.136.184:50292 |
370 | 10.67.188.12:37480 |
371 | 10.67.129.137:41412 |
372 | 10.67.183.216:60678 |
373 | 10.67.164.73:44320 |
374 | 10.67.164.172:51160 |
375 | 10.67.148.86:49710 |
376 | 10.67.131.235:43754 |
377 | 10.67.156.54:43250 |
378 | 10.67.148.125:53326 |
379 | 10.67.139.85:51354 |
380 | 10.67.139.161:41342 |
381 | 10.67.183.34:36416 |
382 | 10.67.131.23:33300 |
383 | 10.67.148.233:38564 |
384 | 10.67.164.179:56352 |
385 | 10.67.165.87:45116 |
386 | 10.67.160.180:33970 |
387 | 10.67.161.184:34416 |
388 | 10.67.191.24:51912 |
389 | 10.67.133.223:56572 |
390 | 10.67.149.255:55558 |
391 | 10.67.152.44:44212 |
392 | 10.67.145.231:53332 |
393 | 10.67.148.125:38802 |
394 | 10.67.154.18:38360 |
395 | 10.67.190.117:58064 |
396 | 10.67.141.53:45128 |
397 | 10.67.172.6:48108 |
398 | 10.67.145.227:40704 |
399 | 10.67.164.73:38002 |
400 | 10.67.129.57:47990 |
401 | 10.67.140.235:51178 |
402 | 10.67.154.147:37154 |
403 | 10.67.161.40:36472 |
404 | 10.67.142.15:43946 |
405 | 10.67.174.132:50068 |
406 | 10.67.164.36:36788 |
407 | 10.67.181.142:56662 |
408 | 10.67.148.110:42156 |
409 | 10.67.158.231:35280 |
410 | 10.67.168.146:47728 |
411 | 10.67.132.238:51102 |
412 | 10.67.163.97:51108 |
413 | 10.67.172.185:35460 |
414 | 10.67.176.184:44572 |
415 | 10.67.138.209:49362 |
416 | 10.67.188.91:42348 |
417 | 10.67.186.161:33174 |
418 | 10.67.148.15:45018 |
419 | 10.67.188.185:59436 |
420 | 10.67.189.158:55168 |
421 | 10.67.157.35:60836 |
422 | 10.67.176.86:47946 |
423 | 10.67.141.42:53662 |
424 | 10.67.191.252:52368 |
425 | 10.67.152.45:33262 |
426 | 10.67.173.201:49996 |
427 | 10.67.163.188:45986 |
428 | 10.67.153.118:41218 |
429 | 10.67.176.64:46486 |
430 | 10.67.190.84:52530 |
431 | 10.67.158.29:41256 |
432 | 10.67.128.221:48008 |
433 | 10.64.0.155:33506 |
434 | 10.67.167.15:52588 |
435 | 10.67.130.94:51852 |
436 | 10.67.162.46:47648 |
437 | 10.67.149.9:44892 |
438 | 10.67.184.170:33730 |
439 | 10.67.154.18:60438 |
440 | 10.67.175.16:44152 |
441 | 10.67.154.50:36254 |
442 | 10.67.146.73:38286 |
443 | 10.67.151.237:43044 |
444 | 10.67.173.244:54278 |
445 | 10.67.128.253:49486 |
446 | 10.67.167.15:41516 |
447 | 10.64.48.14:33618 |
448 | 10.67.142.234:55598 |
449 | 10.67.190.88:33740 |
450 | 10.67.162.46:47662 |
451 | 10.67.156.60:50384 |
452 | 10.67.185.122:56432 |
453 | 10.67.136.54:53014 |
454 | 10.67.133.237:43058 |
455 | 10.67.144.103:51180 |
456 | 10.67.176.64:45916 |
457 | 10.67.147.150:55200 |
458 | 10.67.144.124:52184 |
459 | 10.67.152.36:35090 |
460 | 10.67.130.107:57712 |
461 | 10.67.140.7:46878 |
462 | 10.67.140.87:60470 |
463 | 10.67.176.165:58460 |
464 | 10.67.140.194:46818 |
465 | 10.67.130.12:49404 |
466 | 10.67.160.92:59852 |
467 | 10.67.175.172:39594 |
468 | 10.67.183.187:58670 |
469 | 10.67.183.228:42578 |
470 | 10.67.170.169:39274 |
471 | 10.67.185.142:53138 |
472 | 10.67.188.43:55482 |
473 | 10.67.180.53:34048 |
474 | 10.67.144.189:38950 |
475 | 10.67.180.61:44994 |
476 | 10.67.171.46:44000 |
477 | 10.67.144.124:52190 |
478 | 10.67.162.235:60614 |
479 | 10.67.171.30:43128 |
480 | 10.67.185.206:51238 |
481 | 10.67.157.205:55176 |
482 | 10.67.137.242:38312 |
483 | 10.64.48.192:34480 |
484 | 10.67.180.61:45004 |
485 | 10.67.152.44:32856 |
486 | 10.67.139.27:46614 |
487 | 10.67.135.87:39152 |
488 | 10.67.137.162:42866 |
489 | 10.67.189.158:39000 |
490 | 10.67.148.15:45418 |
491 | 10.64.48.193:33642 |
492 | 10.67.133.157:43872 |
493 | 10.67.173.250:41600 |
494 | 10.67.149.255:50568 |
495 | 10.67.186.213:42476 |
496 | 10.67.143.222:57784 |
497 | 10.67.181.212:37938 |
498 | 10.67.158.66:45396 |
499 | 10.67.173.244:59512 |
500 | 10.67.160.15:60224 |
501 | 10.67.186.203:45532 |
502 | 10.67.172.6:50218 |
503 | 10.67.140.207:60496 |
504 | 10.67.133.174:47344 |
505 | 10.67.129.10:37308 |
506 | 10.67.176.165:42034 |
507 | 10.67.153.118:33838 |
508 | 10.67.185.202:43916 |
509 | 10.67.138.42:60836 |
510 | 10.67.152.197:34174 |
511 | 10.67.147.138:59350 |
512 | 10.67.151.1:56934 |
513 | 10.67.172.139:39998 |
514 | 10.67.140.12:49362 |
515 | 10.64.0.155:38586 |
516 | 10.67.179.93:55136 |
517 | 10.67.148.26:47980 |
518 | 10.67.165.115:58824 |
519 | 10.67.134.50:59498 |
520 | 10.67.190.78:35066 |
521 | 10.67.146.18:56920 |
522 | 10.67.152.45:45180 |
523 | 10.67.151.239:52046 |
524 | 10.64.48.54:41100 |
525 | 10.67.189.202:43356 |
526 | 10.67.151.237:36046 |
527 | 10.67.153.115:49058 |
528 | 10.67.143.99:45226 |
529 | 10.67.181.200:46138 |
530 | 10.67.185.96:49262 |
531 | 10.67.168.146:34992 |
532 | 10.67.139.82:58158 |
533 | 10.64.48.54:41118 |
534 | 10.67.163.97:60844 |
535 | 10.67.150.234:47226 |
536 | 10.67.150.242:58332 |
537 | 10.67.189.158:51304 |
538 | 10.67.154.51:33708 |
539 | 10.67.137.242:35266 |
540 | 10.67.189.158:51314 |
541 | 10.67.158.59:34156 |
542 | 10.67.181.115:52702 |
543 | 10.67.148.86:35300 |
544 | 10.67.159.206:45132 |
545 | 10.67.162.47:49610 |
546 | 10.67.148.76:47172 |
547 | 10.67.153.29:57150 |
548 | 10.194.179.164:54636 |
549 | 10.67.139.41:38390 |
550 | 10.67.134.106:35854 |
551 | 10.67.145.75:50614 |
552 | 10.67.181.179:57612 |
553 | 10.67.138.9:51008 |
554 | 10.67.168.111:38606 |
555 | 10.67.185.211:42936 |
556 | 10.67.175.86:49336 |
557 | 10.67.132.176:49754 |
558 | 10.67.185.96:59634 |
559 | 10.67.180.129:51140 |
560 | 10.64.48.192:37526 |
561 | 10.67.137.19:45302 |
562 | 10.67.147.59:59708 |
563 | 10.67.156.167:50336 |
564 | 10.67.168.37:40572 |
565 | 10.67.185.206:60542 |
566 | 10.67.134.123:37484 |
567 | 10.67.132.174:58430 |
568 | 10.67.137.253:42370 |
569 | 10.67.180.61:50870 |
570 | 10.67.140.207:55674 |
571 | 10.67.133.180:46216 |
572 | 10.67.156.60:52502 |
573 | 10.67.138.167:42770 |
574 | 10.67.152.36:45116 |
575 | 10.67.161.34:39694 |
576 | 10.67.176.184:44904 |
577 | 10.67.184.155:42456 |
578 | 10.64.48.14:46772 |
579 | 10.67.181.142:46410 |
580 | 10.67.170.169:35216 |
581 | 10.67.153.73:41204 |
582 | 10.67.168.160:38488 |
583 | 10.67.134.106:41554 |
584 | 10.67.185.87:50236 |
585 | 10.67.146.30:39464 |
586 | 10.67.148.192:36862 |
587 | 10.67.144.201:45010 |
588 | 10.67.177.160:41700 |
589 | 10.67.139.154:37312 |
590 | 10.67.136.87:34126 |
591 | 10.67.134.106:41568 |
592 | 10.67.184.247:48244 |
593 | 10.67.148.194:45570 |
594 | 10.67.159.243:39640 |
595 | 10.67.146.30:35260 |
596 | 10.67.186.161:33646 |
597 | 10.67.151.185:42666 |
598 | 10.67.140.188:49186 |
599 | 10.67.163.233:40626 |
600 | 10.67.178.85:39894 |
601 | 10.67.162.191:48794 |
602 | 10.67.149.207:51064 |
603 | 10.67.161.156:50332 |
604 | 10.67.183.251:35018 |
605 | 10.67.176.15:35886 |
606 | 10.67.156.191:51846 |
607 | 10.64.48.54:51092 |
608 | 10.67.140.7:33048 |
609 | 10.67.181.179:34636 |
610 | 10.67.191.90:54200 |
611 | 10.67.166.218:57420 |
612 | 10.67.144.230:46730 |
613 | 10.67.183.112:60186 |
614 | 10.64.48.192:50094 |
615 | 10.67.139.41:60672 |
616 | 10.67.136.110:57162 |
617 | 10.67.140.40:40152 |
618 | 10.67.161.143:50000 |
619 | 10.67.185.155:39880 |
620 | 10.67.136.202:44010 |
621 | 10.67.143.20:41930 |
622 | 10.67.167.15:55128 |
623 | 10.67.167.15:55134 |
624 | 10.67.150.87:52934 |
625 | 10.67.183.131:50580 |
626 | 10.67.172.185:54772 |
627 | 10.67.166.67:37538 |
628 | 10.67.170.169:36326 |
629 | 10.67.168.105:33844 |
630 | 10.67.184.25:44818 |
631 | 10.67.183.112:60202 |
632 | 10.67.190.143:46656 |
633 | 10.67.182.172:36830 |
634 | 10.67.148.86:57408 |
635 | 10.67.148.76:49346 |
636 | 10.67.190.47:57516 |
637 | 10.67.128.240:41188 |
638 | 10.67.187.244:36628 |
639 | 10.67.135.96:35680 |
640 | 10.67.169.209:38468 |
641 | 10.67.182.172:36832 |
642 | 10.67.140.147:58504 |
643 | 10.67.163.97:39414 |
644 | 10.67.131.229:38232 |
645 | 10.67.171.30:51878 |
646 | 10.67.157.76:55364 |
647 | 10.67.141.226:58882 |
648 | 10.67.181.104:34154 |
649 | 10.67.153.73:40292 |
650 | 10.67.158.5:50026 |
651 | 10.67.151.221:58420 |
652 | 10.67.162.235:39562 |
653 | 10.67.150.68:60110 |
654 | 10.67.158.233:39802 |
655 | 10.67.162.191:47830 |
656 | 10.67.162.235:39578 |
657 | 10.67.147.133:58570 |
658 | 10.67.191.24:51994 |
659 | 10.67.161.156:55726 |
660 | 10.67.130.69:54118 |
661 | 10.67.182.167:38240 |
662 | 10.64.48.54:33594 |
663 | 10.67.170.169:43804 |
664 | 10.67.166.194:50192 |
665 | 10.67.190.84:48300 |
666 | 10.67.181.142:33534 |
667 | 10.67.128.221:34006 |
668 | 10.64.48.192:58618 |
669 | 10.67.133.174:60388 |
670 | 10.67.186.37:59474 |
671 | 10.67.138.134:54484 |
672 | 10.67.148.86:43192 |
673 | 10.67.164.168:37048 |
674 | 10.67.153.158:39584 |
675 | 10.67.157.76:44138 |
676 | 10.67.149.194:40528 |
677 | 10.67.145.231:50572 |
678 | 10.67.163.188:37568 |
679 | 10.67.156.54:38846 |
680 | 10.67.130.213:41600 |
681 | 10.67.150.240:44830 |
682 | 10.67.135.21:46688 |
683 | 10.67.160.91:38650 |
684 | 10.67.149.194:47544 |
685 | 10.67.131.46:41842 |
686 | 10.67.179.104:56264 |
687 | 10.67.148.220:34446 |
688 | 10.67.143.49:51250 |
689 | 10.67.139.161:52790 |
690 | 10.67.138.131:55966 |
691 | 10.67.145.75:57678 |
692 | 10.67.171.223:57106 |
693 | 10.67.165.116:47760 |
694 | 10.67.183.47:56878 |
695 | 10.67.176.91:39972 |
696 | 10.67.139.5:49052 |
697 | 10.67.180.24:33538 |
698 | 10.67.145.75:57690 |
699 | 10.67.130.254:33778 |
700 | 10.67.186.213:58730 |
701 | 10.67.139.249:42592 |
702 | 10.67.147.29:43446 |
703 | 10.67.133.249:45084 |
704 | 10.67.163.209:57722 |
705 | 10.67.130.30:58196 |
706 | 10.67.130.220:34028 |
707 | 10.67.139.249:42596 |
708 | 10.67.133.180:39226 |
709 | 10.194.132.54:32890 |
710 | 10.67.180.50:38092 |
711 | 10.67.160.189:58204 |
712 | 10.67.161.232:48360 |
713 | 10.67.165.116:35590 |
714 | 10.67.129.5:44330 |
715 | 10.67.154.112:55324 |
716 | 10.67.163.183:52010 |
717 | 10.67.140.87:37678 |
718 | 10.67.150.92:38418 |
719 | 10.194.169.155:54058 |
720 | 10.67.150.72:37274 |
721 | 10.67.138.200:43878 |
722 | 10.67.146.181:50612 |
723 | 10.67.136.229:56738 |
724 | 10.67.172.139:51762 |
725 | 10.67.148.192:43704 |
726 | 10.67.134.181:40998 |
727 | 10.67.179.208:43582 |
728 | 10.67.150.75:34542 |
729 | 10.67.157.40:60962 |
730 | 10.67.132.245:34946 |
731 | 10.67.140.40:53552 |
732 | 10.67.130.254:46832 |
733 | 10.67.132.245:34954 |
734 | 10.67.134.106:45574 |
735 | 10.67.141.42:41818 |
736 | 10.67.154.18:59264 |
737 | 10.67.150.51:39926 |
738 | 10.67.149.255:58004 |
739 | 10.67.163.75:45360 |
740 | 10.67.166.150:50060 |
741 | 10.67.149.207:36352 |
742 | 10.67.133.157:44258 |
743 | 10.67.188.185:45056 |
744 | 10.67.137.222:58554 |
745 | 10.67.136.14:51288 |
746 | 10.67.143.99:52674 |
747 | 10.67.186.161:59464 |
748 | 10.67.180.3:42026 |
749 | 10.67.149.207:36356 |
750 | 10.67.177.214:47278 |
751 | 10.67.186.105:44052 |
752 | 10.67.148.26:60800 |
753 | 10.67.190.14:39428 |
754 | 10.67.149.130:46340 |
755 | 10.67.150.242:33704 |
756 | 10.67.152.44:53384 |
757 | 10.67.148.26:60806 |
758 | 10.67.163.188:54424 |
759 | 10.67.172.30:44684 |
760 | 10.67.187.201:40234 |
761 | 10.67.175.109:51902 |
762 | 10.67.159.23:58302 |
763 | 10.67.162.240:40180 |
764 | 10.67.181.115:41728 |
765 | 10.67.153.139:60552 |
766 | 10.67.154.9:58852 |
767 | 10.67.152.212:38490 |
768 | 10.67.178.142:55222 |
769 | 10.67.145.104:59070 |
770 | 10.67.162.235:43424 |
771 | 10.67.140.77:52668 |
772 | 10.67.157.96:46476 |
773 | 10.67.183.187:40322 |
774 | 10.67.133.223:40840 |
775 | 10.67.133.223:40848 |
776 | 10.67.177.214:54468 |
777 | 10.67.154.18:45224 |
778 | 10.67.181.142:36596 |
779 | 10.67.138.205:45204 |
780 | 10.67.138.241:48126 |
781 | 10.67.176.184:39920 |
782 | 10.67.185.122:38906 |
783 | 10.67.143.2:51202 |
784 | 10.67.132.176:54904 |
785 | 10.67.164.73:32796 |
786 | 10.67.188.111:46876 |
787 | 10.67.156.223:41368 |
788 | 10.67.140.147:37198 |
789 | 10.67.161.232:44104 |
790 | 10.67.161.156:45698 |
791 | 10.67.191.235:59722 |
792 | 10.67.161.34:55494 |
793 | 10.67.161.156:45708 |
794 | 10.67.152.36:52386 |
795 | 10.67.191.126:37256 |
796 | 10.67.185.187:33902 |
797 | 10.67.136.96:35810 |
798 | 10.67.153.210:59046 |
799 | 10.67.161.156:45712 |
800 | 10.67.134.64:36514 |
801 | 10.67.154.123:32808 |
802 | 10.67.160.38:35748 |
803 | 10.67.131.235:33528 |
804 | 10.67.165.115:48922 |
805 | 10.67.148.180:37040 |
806 | 10.67.177.134:37954 |
807 | 10.67.153.73:38390 |
808 | 10.67.168.105:51118 |
809 | 10.67.189.197:45434 |
810 | 10.67.140.77:52678 |
811 | 10.67.151.237:36660 |
812 | 10.67.143.20:34324 |
813 | 10.67.188.63:56456 |
814 | 10.67.159.70:56004 |
815 | 10.67.134.181:36474 |
816 | 10.67.159.70:56012 |
817 | 10.67.191.121:49150 |
818 | 10.67.190.140:53398 |
819 | 10.67.143.2:51218 |
820 | 10.67.150.129:35380 |
821 | 10.67.151.174:56900 |
822 | 10.67.175.16:35160 |
823 | 10.67.134.166:41662 |
824 | 10.67.137.178:35136 |
825 | 10.67.185.206:57468 |
826 | 10.67.156.60:47136 |
827 | 10.67.176.165:48196 |
828 | 10.67.181.142:36600 |
829 | 10.67.161.34:55496 |
830 | 10.67.168.105:51134 |
831 | 10.67.148.180:37052 |
832 | 10.67.142.110:42346 |
833 | 10.67.172.185:52426 |
834 | 10.67.159.70:56022 |
835 | 10.67.148.50:54282 |
836 | 10.67.156.60:47146 |
837 | 10.67.172.185:52442 |
838 | 10.67.128.207:43864 |
839 | 10.67.187.244:54976 |
840 | 10.67.156.60:47156 |
841 | 10.67.179.223:42128 |
842 | 10.67.157.98:58778 |
843 | 10.67.136.54:40754 |
844 | 10.67.164.36:38072 |
845 | 10.67.170.142:44552 |
846 | 10.67.157.96:46492 |
847 | 10.67.181.226:49264 |
848 | 10.67.153.73:38404 |
849 | 10.67.157.20:42946 |
850 | 10.67.149.83:52180 |
851 | 10.67.145.75:50456 |
852 | 10.67.132.32:60152 |
853 | 10.67.183.187:40324 |
854 | 10.67.191.126:37262 |
855 | 10.67.191.126:37266 |
856 | 10.67.150.129:35386 |
857 | 10.67.144.124:37858 |
858 | 10.67.138.1:36346 |
859 | 10.67.151.237:36674 |
860 | 10.67.188.111:46886 |
861 | 10.67.175.172:49036 |
862 | 10.67.128.207:43876 |
863 | 10.67.191.121:49166 |
864 | 10.67.162.26:50634 |
865 | 10.67.160.91:40804 |
866 | 10.67.184.25:54920 |
867 | 10.67.156.223:41376 |
868 | 10.67.175.16:35170 |
869 | 10.67.174.25:47844 |
870 | 10.67.132.174:37092 |
871 | 10.67.169.126:54034 |
872 | 10.67.151.174:56904 |
873 | 10.67.176.184:39934 |
874 | 10.67.129.149:45774 |
875 | 10.67.166.226:38530 |
876 | 10.67.133.180:35598 |
877 | 10.67.189.197:45440 |
878 | 10.67.176.165:48200 |
879 | 10.67.152.219:39702 |
880 | 10.67.148.233:52072 |
881 | 10.67.169.126:54048 |
882 | 10.67.189.161:36786 |
883 | 10.67.171.30:36756 |
884 | 10.67.138.205:45214 |
885 | 10.67.133.237:41360 |
886 | 10.67.148.50:54288 |
887 | 10.67.157.96:44294 |
888 | 10.67.179.223:42144 |
889 | 10.67.162.235:43436 |
890 | 10.67.163.180:58462 |
891 | 10.67.132.176:54912 |
892 | 10.67.175.43:37462 |
893 | 10.67.128.253:42060 |
894 | 10.67.154.18:45228 |
895 | 10.67.132.32:60154 |
896 | 10.67.190.78:36602 |
897 | 10.67.165.115:48934 |
898 | 10.67.170.178:38570 |
899 | 10.67.160.38:35750 |
900 | 10.67.133.180:35604 |
901 | 10.67.172.6:36058 |
902 | 10.67.173.201:53340 |
903 | 10.67.157.40:49902 |
904 | 10.67.168.105:51136 |
905 | 10.67.134.64:45224 |
906 | 10.67.175.43:37470 |
907 | 10.67.166.226:38546 |
908 | 10.67.148.110:38364 |
909 | 10.67.131.46:42436 |
910 | 10.67.190.140:53408 |
911 | 10.67.132.32:60160 |
912 | 10.67.137.102:33896 |
913 | 10.67.175.172:49050 |
914 | 10.67.174.132:41538 |
915 | 10.67.175.43:37472 |
916 | 10.67.156.189:52990 |
917 | 10.67.180.179:51712 |
918 | 10.67.132.32:60162 |
919 | 10.67.139.143:48898 |
920 | 10.67.148.110:38372 |
921 | 10.67.162.26:50644 |
922 | 10.67.178.149:51042 |
923 | 10.67.188.63:56470 |
924 | 10.67.140.207:51108 |
925 | 10.67.144.253:56904 |
926 | 10.67.179.223:42160 |
927 | 10.67.183.100:33152 |
928 | 10.67.190.88:41310 |
929 | 10.67.189.197:45456 |
930 | 10.67.154.123:35790 |
931 | 10.67.136.96:35820 |
932 | 10.67.145.208:33570 |
933 | 10.67.136.96:35828 |
934 | 10.67.168.3:52524 |
935 | 10.67.154.123:35796 |
936 | 10.67.137.175:54870 |
937 | 10.67.143.87:41048 |
938 | 10.67.166.226:38556 |
939 | 10.67.156.189:53002 |
940 | 10.67.145.61:59106 |
941 | 10.67.190.140:53424 |
942 | 10.67.149.255:33908 |
943 | 10.67.158.66:47280 |
944 | 10.67.176.165:48212 |
945 | 10.67.175.57:44922 |
946 | 10.67.148.233:52078 |
947 | 10.67.187.68:42846 |
948 | 10.67.146.89:52138 |
949 | 10.67.128.253:42068 |
950 | 10.67.143.87:41064 |
951 | 10.67.152.191:34528 |
952 | 10.67.175.16:35182 |
953 | 10.67.148.180:37058 |
954 | 10.67.138.205:45226 |
955 | 10.67.133.180:35618 |
956 | 10.67.135.96:34054 |
957 | 10.67.169.254:36860 |
958 | 10.67.134.166:41676 |
959 | 10.67.176.184:39950 |
960 | 10.67.153.158:51888 |
961 | 10.67.130.129:55498 |
962 | 10.67.161.34:55508 |
963 | 10.67.154.18:45240 |
964 | 10.67.153.118:50030 |
965 | 10.67.142.110:42350 |
966 | 10.67.129.137:45336 |
967 | 10.67.138.1:36354 |
968 | 10.67.188.63:56472 |
969 | 10.67.162.170:58574 |
970 | 10.67.178.136:35732 |
971 | 10.67.176.165:48228 |
972 | 10.67.172.6:36072 |
973 | 10.67.135.96:34066 |
974 | 10.67.133.180:35620 |
975 | 10.67.146.89:52142 |
976 | 10.67.162.223:34916 |
977 | 10.67.145.43:54712 |
978 | 10.67.149.180:43566 |
979 | 10.67.145.208:33572 |
980 | 10.67.134.64:45240 |
981 | 10.67.138.205:45232 |
982 | 10.67.154.18:45246 |
983 | 10.67.145.61:59120 |
984 | 10.67.156.60:40344 |
985 | 10.67.137.102:33898 |
986 | 10.67.152.136:55910 |
987 | 10.67.145.75:50462 |
988 | 10.67.176.42:43116 |
989 | 10.67.181.131:44784 |
990 | 10.67.183.131:33724 |
991 | 10.67.148.50:54302 |
992 | 10.67.188.63:56484 |
993 | 10.67.139.206:55150 |
994 | 10.67.160.91:58592 |
995 | 10.67.180.61:37858 |
996 | 10.67.130.18:47120 |
997 | 10.67.154.123:35802 |
998 | 10.67.146.89:52146 |
999 | 10.67.143.2:39682 |
1000 | 10.67.176.184:39952 |
1001 | 10.67.130.12:42656 |
1002 | 10.67.183.131:33738 |
1003 | 10.67.190.140:53440 |
1004 | 10.67.183.187:40328 |
1005 | 10.67.147.59:55908 |
1006 | 10.67.183.187:40338 |
1007 | 10.67.147.150:50854 |
1008 | 10.67.190.23:33326 |
1009 | 10.67.147.1:33962 |
1010 | 10.67.165.75:60218 |
1011 | 10.67.134.2:47898 |
1012 | 10.67.142.234:53772 |
1013 | 10.67.130.69:55680 |
1014 | 10.67.154.39:58266 |
1015 | 10.67.170.146:45952 |
1016 | 10.67.161.77:46716 |
1017 | 10.67.143.84:49764 |
1018 | 10.67.181.118:39808 |
1019 | 10.67.146.174:32994 |
1020 | 10.67.137.19:48652 |
1021 | 10.67.178.149:56352 |
1022 | 10.67.146.189:42308 |
1023 | 10.67.134.64:45252 |
1024 | 10.67.131.226:57020 |
1025 | 10.67.133.56:34780 |
1026 | 10.67.180.221:60020 |
1027 | 10.67.163.52:53910 |
1028 | 10.64.48.192:41322 |
1029 | 10.67.138.42:44958 |
1030 | 10.67.132.176:55602 |
1031 | 10.67.130.142:44186 |
1032 | 10.67.177.143:49852 |
1033 | 10.67.139.154:43984 |
1034 | 10.67.190.251:33724 |
1035 | 10.67.171.219:35098 |
1036 | 10.67.157.117:41428 |
1037 | 10.67.188.111:46196 |
1038 | 10.67.156.22:40528 |
1039 | 10.67.138.200:51142 |
1040 | 10.67.191.24:50060 |
1041 | 10.67.153.66:34016 |
1042 | 10.67.157.76:51858 |
1043 | 10.67.141.217:38246 |
1044 | 10.67.156.223:60050 |
1045 | 10.67.128.240:38612 |
1046 | 10.67.150.75:58854 |
1047 | 10.67.161.143:60362 |
1048 | 10.67.145.221:59476 |
1049 | 10.67.150.92:40072 |
1050 | 10.67.132.180:42438 |
1051 | 10.67.166.151:44546 |
1052 | 10.67.129.15:47456 |
1053 | 10.67.130.81:39326 |
1054 | 10.64.48.192:41870 |
1055 | 10.67.181.212:40716 |
1056 | 10.67.180.254:58716 |
1057 | 10.67.137.253:37430 |
1058 | 10.67.178.180:45972 |
1059 | 10.67.135.189:47076 |
1060 | 10.67.136.14:60056 |
1061 | 10.67.142.45:49164 |
1062 | 10.67.152.10:55694 |
1063 | 10.67.159.24:40218 |
1064 | 10.67.146.189:46582 |
1065 | 10.67.188.43:59658 |
1066 | 10.67.166.194:35660 |
1067 | 10.67.145.208:47972 |
1068 | 10.67.156.191:42606 |
1069 | 10.67.151.19:39380 |
1070 | 10.67.158.206:33966 |
1071 | 10.194.162.215:37406 |
1072 | 10.67.168.105:53446 |
1073 | 10.67.150.51:45040 |
1074 | 10.67.141.217:53926 |
1075 | 10.67.150.234:45572 |
1076 | 10.67.160.214:50978 |
1077 | 10.67.145.227:38896 |
1078 | 10.67.190.23:59942 |
1079 | 10.67.185.87:55950 |
1080 | 10.67.143.193:45776 |
1081 | 10.67.163.188:39870 |
1082 | 10.67.149.65:51840 |
1083 | 10.67.156.175:45168 |
1084 | 10.67.145.104:41200 |
1085 | 10.67.136.14:52008 |
1086 | 10.67.129.79:54172 |
1087 | 10.67.160.180:56140 |
1088 | 10.67.164.5:60794 |
1089 | 10.67.185.202:57926 |
1090 | 10.67.151.45:59140 |
1091 | 10.67.166.194:35072 |
1092 | 10.67.162.183:44804 |
1093 | 10.67.130.254:38010 |
1094 | 10.67.190.209:33058 |
1095 | 10.67.183.34:59802 |
1096 | 10.67.157.35:59598 |
1097 | 10.67.176.64:45368 |
1098 | 10.67.176.22:55462 |
1099 | 10.67.159.243:34898 |
1100 | 10.67.132.238:54688 |
1101 | 10.67.129.57:43798 |
1102 | 10.67.190.143:46144 |
1103 | 10.67.133.244:37624 |
1104 | 10.67.160.118:38790 |
1105 | 10.67.136.87:37640 |
1106 | 10.67.153.135:34040 |
1107 | 10.67.163.209:45806 |
1108 | 10.67.139.255:36132 |
1109 | 10.67.184.170:33440 |
1110 | 10.67.158.212:53128 |
1111 | 10.67.135.170:50160 |
1112 | 10.67.140.244:37350 |
1113 | 10.67.148.147:35882 |
1114 | 10.67.174.11:58288 |
1115 | 10.67.175.113:39268 |
1116 | 10.67.145.62:45580 |
1117 | 10.67.175.172:55918 |
1118 | 10.67.185.206:58498 |
1119 | 10.67.189.202:44748 |
1120 | 10.67.188.43:39616 |
1121 | 10.67.181.179:45260 |
1122 | 10.67.148.125:34644 |
1123 | 10.67.159.94:33658 |
1124 | 10.67.190.114:32790 |
1125 | 10.67.151.221:33142 |
1126 | 10.67.182.135:50740 |
1127 | 10.67.185.206:45984 |
1128 | 10.67.135.96:46984 |
1129 | 10.67.180.228:52974 |
1130 | 10.67.174.25:45432 |
1131 | 10.67.181.142:39282 |
1132 | 10.67.162.46:47098 |
1133 | 10.67.152.191:44562 |
1134 | 10.67.190.78:35320 |
1135 | 10.67.170.178:45828 |
1136 | 10.67.142.110:53626 |
1137 | 10.67.183.100:41516 |
1138 | 10.67.162.170:59500 |
1139 | 10.67.148.50:47566 |
1140 | 10.67.188.63:47576 |
1141 | 10.67.129.137:52432 |
1142 | 10.67.188.111:45252 |
1143 | 10.67.157.96:53610 |
1144 | 10.67.162.26:55432 |
1145 | 10.67.181.226:59324 |
1146 | 10.67.172.185:46694 |
1147 | 10.67.132.176:37050 |
1148 | 10.67.183.187:50908 |
1149 | 10.67.145.45:56326 |
1150 | 10.67.177.134:37080 |
1151 | 10.67.189.161:39994 |
1152 | 10.67.172.243:53864 |
1153 | 10.67.139.154:52244 |
1154 | 10.67.153.118:59898 |
1155 | 10.67.170.178:45842 |
1156 | 10.67.166.194:33060 |
1157 | 10.67.139.154:52258 |
1158 | 10.67.136.149:46528 |
1159 | 10.67.143.193:49128 |
1160 | 10.67.149.180:34278 |
1161 | 10.67.162.170:51300 |
1162 | 10.67.162.26:33822 |
1163 | 10.67.129.162:56582 |
1164 | 10.67.147.150:40380 |
1165 | 10.67.161.232:55352 |
1166 | 10.67.179.223:56856 |
1167 | 10.67.181.104:49496 |
1168 | 10.67.178.142:36432 |
1169 | 10.67.163.200:42198 |
1170 | 10.67.143.20:53362 |
1171 | 10.67.142.15:40898 |
1172 | 10.67.153.29:48044 |
1173 | 10.67.164.5:60084 |
1174 | 10.67.131.152:53752 |
1175 | 10.67.188.111:53022 |
1176 | 10.67.164.73:51702 |
1177 | 10.67.148.50:47742 |
1178 | 10.67.166.151:51622 |
1179 | 10.67.138.167:59290 |
1180 | 10.67.131.46:38196 |
As this happened again tonight, @Mahir256 reached out to me in private and pointed out that most recent edits on Commons before the outage seemed to come from Cat-a-lot (edit: also previously mentioned at T370304#10065664). Looking at its source code, I notice that it makes edits without the maxlag parameter; it’s allowed to do that according to the API etiquette because it’s an interactive tool, but maybe it should do that use maxlag anyway, so it’ll automatically back off if replication lag increases too much? It can apparently produce substantial numbers of edits (it’s the #2 tag on Commons, only slightly behind SDC bots.)
(Note that users will definitely notice this, and it will result in some lost work because I don’t think Cat-a-lot has strong support for retrying failed edits later – as far as I can tell, it just retries up to 4 times after waiting for 300 ms each time, which is probably not enough for replication lag to significantly go down. As it happens, volunteer me has also written QuickCategories, a tool that can edit categories in batches which are saved more permanently and which automatically retries failed commands after a longer delay, and which also uses maxlag of course; maybe asking some of the Cat-a-lot power users to use that tool instead would also help. But all this assumes that Cat-a-lot and its lack of maxlag are related to the problem at all, which is pure speculation so far.)
Closer inspection of Cat-a-lot also reveals that it makes all of its API requests in parallel, rather than waiting for one to complete before starting the next one as the API etiquette suggests. This should be fixed, but it’s not easy given the code structure :/
I noticed that as well. Cat-a-lot can absolutely produce massive edit counts at massive edit rates, especially considering the high rate limits for most users and effective lack of rate limiting for autopatrolled users on Commons. This ratelimit was increased in T194864 specifically to accommodate the use of tools like Cat-a-lot and VFC, so it's absolutely something people are sensitive to. As @Lucas_Werkmeister_WMDE noticed while I'm typing this, Cat-a-lot is old and isn't especially easy to work on, especially since it hasn't had an active maintainer since 2019. It also doesn't handle failures well at all, but I'd say having to batch another 200 files is less annoying than bringing down the cluster, so adding maxlag at some reasonable value might be a good idea while other causes of database load are investigated.
Since cat-a-lot doesn't make edit sequentially and just unleashes all of them at once (e.g. 200 edits at once), it is a violation of API etiquette and very likely the reason behind all of these issues we have had in the past couple of months. The emergency action to make edits serial should stay for now and we continue monitoring to see if it reduces the issue or something else is also a contributing factor here.
very likely the reason behind all of these issues we have had in the past couple of months.
With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.
(data from https://quarry.wmcloud.org/query/85603 )
I guess there's no way to correlate the queries we've captured to Cat-a-lot right? It just makes generic writes?
Random note: The gadget currently waits 1 second before next edit, I think it's fine to lower it to 500ms.
very likely the reason behind all of these issues we have had in the past couple of months.
With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.
These are not mutually exclusive. A bad pattern can feed into another change and cause outages. None can do harm on their own but it doesn't mean they shouldn't be fixed. An example that comes to mind is that due to some bug, update.php was being run on production every night for years, that didn't cause harm until due to other issues a run dropped one of the most important tables we have in production. It doesn't mean running update.php in production was fine because time-wise it doesn't correlate (not to mention cat-a-lot is violating API etiquette and should be fixed regardless). Other reasons that can trigger this to happen is that every edit might get more expensive (new features, T372718: Translate extension should not automatically switch to querying master in all non-GET requests, etc.), it can be that baseline of edits has increased, it can be that people use much larger batches instead (which your graph can't show) or usage pattern has changed, it can be that newer mariadb is less tolerant of large spikes (as it's not designed to), it might be that due to other changes, these slow downs cascade further, and so on and so on. In an environment as complex as ours, we can never be sure and that's why there is an etiquette.
Noting that the rate limit for autopatrolled or other trusted users is way too high, 10500 edits in three minutes translates to roughly 60 edits a second (at the most gentle version that the user doesn't do spikes of 10K edits in second, wait three minutes and then next batch). Our recommendation for bots in Wikidata is one edit per second. The ratelimit should be cut to half at least.
I guess there's no way to correlate the queries we've captured to Cat-a-lot right? It just makes generic writes?
The pressure it puts is the general writes, that's true but we can measure edits done by the gadget through change_tags.
TheDJ added a comment.EditedAug 19 2024, 1:47 PM
very likely the reason behind all of these issues we have had in the past couple of months.
With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.
These are not mutually exclusive.
Yeah in order to scale with the continuing complexity, MediaWiki has made a lot of optimisations in the past years. I'm not at all surprised that something like this could happen.
And let's not forget that up to a few weeks ago, Cat-a-Lot for search results was broken for many many months (ever since the switch to <figure> ????), that is a very long timeframe for 'MediaWiki' to change in a way that nobody was noticing.
Additionally, it only takes one user becoming 'very active' and adopting a process where they use the '500 per page' option, where previous active users were using the 50 items per page view, and you might suddenly tip the bucket.
Browsers do limit the amount of concurrent connections per domain to about 6, so it's not as if all these 500 will come in at once (I think, we might want to double check http2 multiplexing on this detail?). There will simply be a continuous 6 parallel streams of connections at full speed, all leading to deferred async load that needs to be handled at some time. When there is no backing off, and especially when DB rows are 'close' together (not unlikely with categories and a single user doing the actions), I can easily see how that would lead to lock contention. It used to be that every connection would not return until almost all the work was done, so there used to be a bit of a natural brake on streams like this, but nowadays there is a lot more work that happens after the request completes (for very good reasons btw).
So many elements come together in scenarios like this.
Random note: The gadget currently waits 1 second before next edit, I think it's fine to lower it to 500ms.
Somewhat tangential, but enWS often needs to delete up to 1k+ pages in one go. To deal with this we use a gadget that chains $.Deferred() ($.Deferred() instead of Promise so one failure doesn't break the chain) to sequentially delete via the API. This approach should be applicable to Cat-A-Lot too.
PS. The MassDelete Gadget would be a lot less hacky if the Action API supported deleting more than one page at a time, and supported supplying the pages via a generator. Less wacky code to manage client side and MediaWiki could be in complete control of how fast to execute the deletes and with how much parallelism. That probably can't easily be applied to Cat-A-Lot, but for other use cases like deletes it would be a lot cleaner all around.
DB rows are 'close' together (not unlikely with categories and a single user doing the actions),
It's actually the same rows being updated and causing contention: it's updating user_editcount for the same user and it's updating category membership numbers for the same categories so it's quite likely they would deadlock or cause contention (as most locks in databases are row-level). These two were the ones we saw the most contentions.
PS. The MassDelete Gadget would be a lot less hacky if the Action API supported deleting more than one page at a time, and supported supplying the pages via a generator. Less wacky code to manage client side and MediaWiki could be in complete control of how fast to execute the deletes and with how much parallelism. That probably can't easily be applied to Cat-A-Lot, but for other use cases like deletes it would be a lot cleaner all around.
Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).
Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).
Have we ever considered a mechanism for gadgets/userscripts/other JS to queue some types of jobs?
For the record, we reduced the rate limit for trusted users to 1500 edits every three minutes which is still quite high but better than 10500 every three minutes: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1063837
Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).
Have we ever considered a mechanism for gadgets/userscripts/other JS to queue some types of jobs?
That's a great idea, at least we should be able to accommodate this for admins for example. Someone needs to just build it :)
It's actually the same rows being updated and causing contention: it's updating user_editcount for the same user and it's updating category membership numbers for the same categories
I wonder if there should be an edit API flag that disables updating aggregate stats like those, and instead queues a job to recalculate them. Recalculating something like edit counts isn't cheap, but probably still cheaper than making a thousand +1 writes.
The more efficient but much more complex approach would be to handle them like site stats: instead of writing the field immediately, storing deltas in a dedicated table and batch processing them.
Have we ever considered a mechanism for gadgets/userscripts/other JS to queue some types of jobs?
That would mean a significant amount of time passing between the user or client-side logic taking action and the action actually happening, which would make it very hard to determine if the action is still valid (what if someone else edited the page in the meantime?), and it would be hard to tell the user when the edit is discarded.
What would probably work better is queueing the editcount etc. updates specifically. These are already deferreds, and there is already a mechanism for converting deferreds to jobs, so that should be relatively straightforward, other than when to trigger it.
For the record, we reduced the rate limit for trusted users to 1500 edits every three minutes which is still quite high but better than 10500 every three minutes: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1063837
That doesn't affect admins, right? How much is it going to help in practice, given that frequency of editing tends to correlated with privileges?
@Marostegui Given that the issue has been identified (and only trusted users have/had the higher rate limits), can this be made public again?
@Marostegui Given that the issue has been identified (and only trusted users have/had the higher rate limits), can this be made public again?
I was planning to do so by the end of the week if we don't have more incidents.
CDanis triaged this task as High priority.Aug 21 2024, 6:16 PM
I am going to close this as fixed for now. We've not had any issues since Sunday. Thanks a lot to everyone who's helped to get fixed, it's been a difficult one. I've asked @Aklapper to make this task public as I don't have rights for it.
With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.
I recall years ago that cat-a-lot, while not taking down the site, still had enough of an impact that you could see when someone was using it on site performance graphs. I'm honestly not that surprised it eventually caused a real problem.
Noting that the rate limit for autopatrolled or other trusted users is way too high, 10500 edits in three minutes translates to roughly 60 edits a second (at the most gentle version that the user doesn't do spikes of 10K edits in second, wait three minutes and then next batch). Our recommendation for bots in Wikidata is one edit per second. The ratelimit should be cut to half at least.
For historical context, the rate limit is high essentially because the commons community demaned a high rate limit for this specific tool.
In T370304#10073278, @TheDJ wrote:
It used to be that every connection would not return until almost all the work was done, so there used to be a bit of a natural brake on streams like this, but nowadays there is a lot more work that happens after the request completes (for very good reasons btw).
Perhaps we should disable deferred updates in API requests? At least per default, we could enable them for certain modules or for users with specific rights.
Api:Etiquette states: Making your requests in series rather than in parallel, by waiting for one request to finish before sending a new request, should result in a safe request rate. But as TheDJ points out, deferred updates lead to a situation where this is no longer true.
In T370304#10073278, @TheDJ wrote:
It used to be that every connection would not return until almost all the work was done, so there used to be a bit of a natural brake on streams like this, but nowadays there is a lot more work that happens after the request completes (for very good reasons btw).
Perhaps we should disable deferred updates in API requests? At least per default, we could enable them for certain modules or for users with specific rights.
Api:Etiquette states: Making your requests in series rather than in parallel, by waiting for one request to finish before sending a new request, should result in a safe request rate. But as TheDJ points out, deferred updates lead to a situation where this is no longer true.
That wasn't really the problem here. Yes. deferred updates will be handled on the server-side while the client has sent the next request but given that server side first needs to be done with the main request, there is enough delay to reduce lock contention time and even if there is a lock contention, it's minimal since the deferred updates form a nice queue. The way Cat-a-Lot was operating was not at all like that, it was unleashing all 500 requests at the same time and was bound only by browsers having a limit of concurrent connections but with HTTP/2 it might have been way higher. We had simultaneous flood of 500 (or more) conflicting requests in an extremely short period of time and that brought down wikis. immediate serial requests might not be very nice but there is no way that they can bring down wikis. That being said, I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.
I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.
Is there an easy way for Gadgets and user scripts to get at maxlag? Does mw.api client-side handle maxlag, or help the coder handle maxlag in any way?
And my point up above somewhere about moving some logic server-side (in my case deletes of multiple pages in a single request, for Cat-a-lot that'd be edits) is that on that side you can more easily serialize operations, rate limit them, have configurable max concurrency, etc.
The facilities available to Gadget and user script developers are extremely limited (we can't even use async/await yet) so brute force approaches are to be expected.
I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.
Is there an easy way for Gadgets and user scripts to get at maxlag? Does mw.api client-side handle maxlag, or help the coder handle maxlag in any way?
You can get maxlag via queries like this: https://www.wikidata.org/w/api.php?action=query&format=json&titles=Main%20Page&maxlag=-1
And my point up above somewhere about moving some logic server-side (in my case deletes of multiple pages in a single request, for Cat-a-lot that'd be edits) is that on that side you can more easily serialize operations, rate limit them, have configurable max concurrency, etc.
Having an API gateway would help here (a bit) but this really the responsibility of the author of API consumer. See https://meta.wikimedia.org/wiki/API_Policy_Update_2024
The facilities available to Gadget and user script developers are extremely limited (we can't even use async/await yet) so brute force approaches are to be expected.
I honestly think we should start allowing async/await, From my understanding and https://caniuse.com/?search=await there shouldn't be any blocker for it now. Considering https://www.mediawiki.org/wiki/Compatibility. Some other frontend dev might chime in and correct me if I'm wrong.
I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.
Even better, vary the wait time based on how long the previous API call took to complete. As long as calls are completing quickly, you keep firing off the next call just as quickly. When the system gets bogged down, you back off. There's a ton of variations on this, but the basic idea is widely used in congestion control algorithms in many applications.
So you have to parse it out of the returned error block? And only if it errors due to exceeding the maxlag value you provided, so you'll need an extra Action API call to get at it? I think maybe there's some scope for the mw.api client-side module to make this a little more accessible to on-wiki developers…
As a reference, I believe Pywikibot just transparently honors maxlag so most bot writers or users of the canned scripts don't even have to think about it. If mw.API did that for Gadget writers too it could help.
And my point up above somewhere about moving some logic server-side (in my case deletes of multiple pages in a single request, for Cat-a-lot that'd be edits) is that on that side you can more easily serialize operations, rate limit them, have configurable max concurrency, etc.
Having an API gateway would help here (a bit) but this really the responsibility of the author of API consumer. See https://meta.wikimedia.org/wiki/API_Policy_Update_2024
Don't get me wrong, I'm not trying to absolve Gadget authors of responsibility for using the Action API endpoints in a sane way, but cf. the massdelete Gadget I linked previously, the hoops we have to jump through to do so are rather excessive. That Gadget could have been a single Action API call (with a generator) or a small number of calls (with a list of pageids) sent in serial inside a simple foreach loop. Instead it involves deeply recursive call chains, factory functions, and jQuery $.Deferred to replace native Promises (most people can't even tell you the difference between those two). You're not going to take the Deadpool approach ("Maximum effort!") unless forced to, so maxing apilimits with 500-1000+ concurrent requests is the most natural and obvious way for Gadget writers to go.
Make it easy to do the right thing, either in the API itself or in the access library (mw.api), and people's lazyness will work in your favour. The alternative is expending the effort on running around and policing everyone that does it "wrong".
The facilities available to Gadget and user script developers are extremely limited (we can't even use async/await yet) so brute force approaches are to be expected.
I honestly think we should start allowing async/await, From my understanding and https://caniuse.com/?search=await there shouldn't be any blocker for it now. Considering https://www.mediawiki.org/wiki/Compatibility. Some other frontend dev might chime in and correct me if I'm wrong.
This would be T357197 and, more broadly, T277675. And, yes, giving the community the option to, and trusting them to, make good tradeoffs with opt-in async/await would be good. I haven't looked at Cat-a-lot's code so I don't know if it'd help any there, but I have an inherited default Gadget that does 3 Action API calls for every single page load on the project that it would simply be too painful and time-consuming to rewrite
without access to async/await (well, and T358660 would also really help that use case). Saying that the functionality the Gadget provides is only available to contributors with browsers that support async/await would be entirely reasonable when held up against reducing needless API calls by up to one order of magnitude.
Whether raising minima for core makes sense is somewhat above my pay grade, but from my perspective it would certainly seem so. It makes no sense for MediaWiki to go to such extreme lengths to preserve better backward-compatibility than standards organizations (W3C, ECMA, etc.) and browser vendors actually enable you to do. It's a naïve approach that ends up pretty much just wasting resources despite its good intentions.
But I'm a ways out in the weeds relative to the core issues in this task so I'll shut up now.
For gadgets using mw.Api() (which is hopefully most of them), the nice solution would be to add a new constructor option to pass a scheduler. Then within mw.Api, we'd just have to change mw.Api.ajax() so that it gets a promise from the scheduler and chains the API call to that, and in gadgets you only need to make some trivial change like new mw.Api() -> new mw.Api( { scheduler: mw.Api.defaultScheduler } ). The scheduler could then decide whether to sequentialize requests or limit them to a fixed number of connections or inject delay or whatever.
So you have to parse it out of the returned error block?
Yes, that's what pywikibot does too.
And only if it errors due to exceeding the maxlag value you provided, so you'll need an extra Action API call to get at it?
Yes, but you can keep the same value during the gadget run. So you don't need to request it before every API request, specially for gadgets. It's also quite cheap to call.
I think maybe there's some scope for the mw.api client-side module to make this a little more accessible to on-wiki developers…
Indeed but e.g. cat-a-lot doesn't wait for the response so any action we do here won't make an impact there. The problem is how javascript handles asynchronous calls vs python which is much more synchronous when it comes to API calls. await/async helps a lot here but first we need to get it out of the door.
https://www.mediawiki.org/wiki/Manual:Maxlag_parameter says that you can add e.g. &maxlag=5 to all of your requests, and retry if they fail with the maxlag error code. You don't need a separate request to fetch the current maxlag value. And you can read the suggested delay to wait before retrying from the Retry-After HTTP response header.
We could add something making use of that to mw.Api. It could be similar to how postWithToken() already works (that method implements retrying on specific failures).
https://www.mediawiki.org/wiki/Manual:Maxlag_parameter says that you can add e.g. &maxlag=5 to all of your requests, and retry if they fail with the maxlag error code. You don't need a separate request to fetch the current maxlag value. And you can read the suggested delay to wait before retrying from the Retry-After HTTP response header.
We could add something making use of that to mw.Api. It could be similar to how postWithToken() already works (that method implements retrying on specific failures).
That would be nice. I'd need to be pretty hard up before I implemented retrying and pulling intervals out of HTTP headers in my Gadget code (the amount of added complexity relative to what's inherent for any given little Gadget does not compute). Now I've been reminded of it I'll probably go add the magic value to my API requests, but anything that errors out is just going to get thrown to the user. Retrying, backoff, etc. belongs in the access library code; not spread out in various half-baked independent implementations cut&paste-propagated across hundreds of Gadgets and user scripts.
Or put another way, when I use mw.Api, make it easy for me to be a good API citizen and make me jump through hoops to not be a good citizen. The status quo is somewhat the opposite.
https://www.mediawiki.org/wiki/Manual:Maxlag_parameter says that you can add e.g. &maxlag=5 to all of your requests, and retry if they fail with the maxlag error code. You don't need a separate request to fetch the current maxlag value. And you can read the suggested delay to wait before retrying from the Retry-After HTTP response header.
That's actually IMHO outdated and needs to be revisited. For example, in Wikidata we ended up in a situation that bots edited so fast that maxlag would go up to above 5s. Then all bots stopped allowing for slow recovery, then once it got to below five seconds, the flood gates opened again and on and on. This built a visible cyclical edit pattern. The idea put forward back then was to build a metric using PID controllers, advertise that value as Retry-After value in all requests (the specs allow it for 200 responses too, I can't find it in RFC 7231 though, I will check but I'm sure we checked the specs) and force bots to respect it in every request. We never got to implement it though.
… the specs allow [Retry-After] for 200 responses too, I can't find it in RFC 7231 though, I will check but I'm sure we checked the specs …
HTTP doesn't generally limit what header fields you can send with a given status code. Both status codes and header fields are extensible, and implementations are required to ignore and pass on any field they do not understand. So at worst the semantics are undefined outside the context of the MediaWiki Action API and its clients. And if you absolutely need to, inventing your own custom header field for this would be entirely ok from a standards perspective (implementation and maintenance perspectives might differ).
PS. RFC 9110 is current, but its description of Retry-After is identical to the one in RFC 7231.
HTTP doesn't generally limit what header fields you can send with a given status code. Both status codes and header fields are extensible, and implementations are required to ignore and pass on any field they do not understand. So at worst the semantics are undefined outside the context of the MediaWiki Action API and its clients. And if you absolutely need to, inventing your own custom header field for this would be entirely ok from a standards perspective (implementation and maintenance perspectives might differ).
HTTP doesn't, but browsers do (with CORS). There are ways to fix that again, i'm just pointing out that adding headers requires more thought, then just slapping them in the response.
I made a new ticket for fixing cat-a-lot.js (T375355) and there is now parallel edit version which limits the editing to 5 parallel edits and 1 if the maxlag is high.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits