2.6.24-rc6 reproducible raid5 hang

This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.

--1171294248-886438723-1198775182=:5207
Content-Type: TEXT/PLAIN; charset=US-ASCII

hey neil -- remember that raid5 hang which me and only one or two others
ever experienced and which was hard to reproduce? we were debugging it
well over a year ago (that box has 400+ day uptime now so at least that
long ago :) the workaround was to increase stripe_cache_size... i seem to
have a way to reproduce something which looks much the same.

setup:

- 2.6.24-rc6
- system has 8GiB RAM but no swap
- 8x750GB in a raid5 with one spare, chunksize 1024KiB.
- mkfs.xfs default options
- mount -o noatime
- dd if=/dev/zero of=/mnt/foo bs=4k count=2621440

that sequence hangs for me within 10 seconds... and i can unhang / rehang
it by toggling between stripe_cache_size 256 and 1024. i detect the hang
by watching "iostat -kx /dev/sd? 5".

i've attached the kernel log where i dumped task and timer state while it
was hung... note that you'll see at some point i did an xfs mount with
external journal but it happens with internal journal as well.

looks like it's using the raid456 module and async api.

anyhow let me know if you need more info / have any suggestions.

-dean
--1171294248-886438723-1198775182=:5207
Content-Type: APPLICATION/octet-stream; name=config-2.6.24-rc6-neemlark1.bz2
Content-Transfer-Encoding: BASE64
Content-Description:
Content-Disposition: attachment; filename=config-2.6.24-rc6-neemlark1.bz2

QlpoOTFBWSZTWcADSgoADBZfgHAQXOf/+j////C////gYF78eQAAAAAAAee+
x8odGo7gW7QZ2Dpo01QetCR6ADQUAAAAUp6vVPo9XO7rkVLQYD77w3eT6MgA
AAAts8UTq62stTn1Va59D66S9R295vd7uvrV97urKTI1iOxQpl0dObKa99Z0
J7Nt26uo+g7jR22ltp3NGgOuPvvrvKiK+vSdbTlojXOgzdMmTdin0YbvXe0p
TVKp973H3YOEep2+kh22xkTKC2tlWyt99uvVrdmVm50NFbXXXayNd26t5U+z
vffPO+9272h1vescr2GpiAAIAIImkxUR6mT9SNP1IyeoNNNAA00CAQIIEEyp
sSaGgAA0AAAYhCZBBpGp6GlPaTCT1TxNqngaak9MKemkNGgk0khNBGgp6EKm
9TU2UGQA0MjRk0AAESIhNCaNBNNU20SZDCA0AAABoAkRAgBBGiCNGlMo8pPU
yMAAANA9T/Sh/+/5xeqf+kluVx1a6VBdZXtmtC/ozbGj9Wqi5tccQUqIOFLE
ZUo4W3MzLjjxozGVXHEHKWqt1hmElVcv62R+aeUOvau0S00n8ZpHCCh61mM8
v7261gtJE/zAKjI5+u5YJpoQmbCQMxFX+1Kz+31zDTw/zppBQelqQ/nT3hOm
vNSnxg2gmWI20rbbWNttS1NJjlLFR+s+vbRsjE01qAiK1LTfKY45RDMGlKli
Zly1URo21RLRahbZSq4mOWjQsylhlUWFu7mUlEFhRpbBtol4zBxp+7ew1pGp
RotKLYKsRRTMzBWlpVLVQU31mtVolRalC7WuDaVtlqItWjS6twtqiVaVAWiV
ayW21LUVzKgpMMISVkCaQkMFmC6GJQWsqxYwuUo3Czev2y4876B0V/gksgsM
rVFq2u2rmrRiq2teMKGNVqVLG22waW5jhVcaylbW/vpY6tNFbRHbG61KmhOX
GQHV1FWILFbZLLaM3cwqJpFtVEuZTLa2tERYsAWLIsBZG0aNymKmFsxuRayX
KSVbVjbF45xcJNIsRFFkjtZMZDGYimMKNWKq4qXKW1vbVMHQpbVxKY20Wgxj
UKLa2qqrUrKNRrWVFsvWhkSK4iqChQFQSCJrLgLjViQo1ylW/rkMVO1111XB
GjvgpKwgiiAiyuGW3EbRctkS2CwRERVrLFywvFxxtNnBZtZrJAriFZK2KmYV
yUkUFkQVxClWmIXC1bLlqoquNVlGqFaGJ1pmNsqAsgokxsXTWKKilYXTGvWm
OUsQS6yXtTTI5dZeLpzSY1UucJcqhaLU5Ycpu7bNMWlKhWH6t0XjKjBywf7h
TftTSTOe2aeGytbbtaI2wIYGVYouNGKY7pjFMDVrK1kmDWjWFTBrHLKyRZ6M
gVkmWzEljNIoaYFZVylcaKgqoxQxMctyGWottgsiWyUtzJjXMQqLHTwmtbYF
UFXGpjUTHw5iVlbZQUurjIVMRhaZiNkB8OJcDBxFCtE3YVytNkxMZcuDy7b7
myKsOybSThCEabOJt4uzHVYjqlSlaUWKY0tIYhjFAWVhKiJiAViytSsqFY2l
ZlSxGxKjgtctzrTBRmKXTTBQqwLrMFEyxpRYCqgOmYwxxtpVClKoW1rLW21b
aS3ArgVcpiXKFYGDIOZkDEgpBYGZiBhWVxhcoY0SVkxJVYOUWBgqKGIGMCoo
YJJlsTGUK41kzbWBjq1xzWvTWM2YYCISDMJOyLoMjxd+Yt6M1tbw+NHwHfxe
fV9pGm/Onf/E9H745xCbNvYs0l49nocgAEmXUGpW5+sC9NzMm/0QBxOxZIdV
/PCskHyqUsoI/ggD43WT+zKKAdbt8mr10J5a+e2RKCzG15dmXtGrP2ah5ZXl
ZPlreLIQ5PCTcLMf6QvRLoNxsRbOuc5TQohRYuxPchi7RJgL9TZVYGihrZxC
EIkVQrp/u5Mgjya+X/l1eq6i7ZsJrwTBtXBa+YSzuP0lRv87OrjjOlRrXTu0
6fL+Ga4sdfH+dF2JXsEpXJNQFAId5U3QtxX/Xy+Ai15lNf9IFnF8SqOSrzSN
cVb/Mi+xK9oY/yLb8m0v0TC+EazzAf4/6Xh7zSbfD9eqcPjdKs/C13VOWGlv
tOemiw++u2yWrpunqP4ph++jvCbpp7DeP+1KC9dv5CxZH/tO/P938O/iPCZB
bIrhKUn+Wly7a9XPuD4jKK2wEcpUPywPbGj0yNPL0DyBSKfULVi3XjP+3MaX
haFFyX2zlvXmO+OHSfWrl6eiMk22WWDIh5rTDQzaLuVj3leD4OZSrr7J3pQs
timhLbhnd9H13PQPiIVVqt6TDZlkdqLzMP82+G1Z9U77Yz5gW7O74KMbX3JA
uJ42Ri9pxhfy0MdIz0yPBSa4NPNiPcGeIvhPWPO2qGeT1kUlXgu9QZN5RoK+
bBkuXlY3W2Z7z5dGmjlPDK1uQYtE3SnU2Qvha4vJq5i/Z3pje8/z/R9fn/Y/
8Iwvw/19IJJBPv7q+q/h6Xt0Vn5Xy5P5amzHD34jn1/fHKwJJBMsbOyL98wY
3M1l/ZDH435CeZM/d1I6rfDRx4+rf3jX9/zRn8f5ED6AgEEn/GP28ESPbzD9
u7+M8juXJz8l+n8ZfmhLh/YQKr0P/v6ePm+G8Nq3759/k3z9K49b7t/Wtvv2
ikcY09EUHN1fsusn8kuhMkh60yKDfHlfagyEcjAuEFIMZMwMh2yHirH3mJJ+
0ElqSWwOF3siF+SP3mNIZk88RpEVwwpotDf6y0guuZsrCIvYMyoigRQlnKmB
ajGjqqugl4gpc/xv/z8s4D1zWGhmfOKNSkhniEcAAsqqBT8wu2lBdJ4V7GwZ
om0sZVmlmXDt/hytHpZ6hU/rN2RxsVMlTXGmFbwWINqBqFyUDWd5gCP0uyXe
xqKlXhypQwtg16SatIWtwansaCRQi1IvL1mLK0zeBW95qpSk2u6jkyyARQop
hpYtFnhl7xlYzkwTAQyLTahVi0pAOlbOIKM3cXcQNh9m7kBjHHhPrGIofr5A
vZ27YYJWAj4pLDHdZwY0z/P4Pc6glZQP+pr03hvIcLFF91Nvig/X/JWdba+T
P0zRspH29dzDBtNlGrNTEB7X3WnP19r28sH7izxtWYCDX+/+ZXmM31WYML+T
z3RgBAe+fX/T79oFIQEwr75U32+Xiqm32EcrfGRyHzDVFqgAv36f74MUUnw2
340YkdGiee6pn1/OqLIynlt/5by3b3ywfeXyHkrSRgufv9+nOJ5UXF/xm58y
zOcV0jyjfkqu5P3m5kaV1iKErcmvj/1+LFdgwVXblYltp54Iyrm0XDkbFaZN
hADBVVhiuEZIBzPGUQQhSCk2lIb1f8W7NXkz/KV66OjVjwHRIDeZy0c95PFz
IQRI3AZVeS5uenHoUcqvjf9Y5TWilsoMtY0X+E9OngjSqhBCB1DUPWInRYwO
9wmvtcgjPy767c4RtNYssf3TGa3j5GhAq7bEKx+DxNVzAUUngQcIdMAtAwkD
GfgdO7bdoZgc3GyunxWA8iabyG5VInBwzJei0+qvn5P89RCaw+AuFZ+x41iG
6nidBx99NibRolYRnugyqw8ukryYkVJ2TxybgIqFwiKIlBiGFACB1gGfrb9t
YBQbvN+6laeNd02K6yIa0iEnmUOu7UKm30816ClhMaZaExv6zm6DGWZH5KTg
2CQqTwTUugt2vxSUcoEZQtveNoUEy9gfLRZ2f5E2ykY8+Ke+sJp4QLWUIRIg
b3Aut6wi54JT8XXo3dXz88gLIXvSZ4GRYh6APHgzb3zQnfM4QAHUfGsSifar
Kc299xk+otc+o8ZrC7yWcVy5NJieF6sxsqdLnMgShU589IKNlooBc9jERVkM
jCuLiCabmqsQD5IKY+HjTUjBgtNvhbqu/bPiTGDkeig4qB3A3eYZjcwvggSo
evQ8qFN3m7mn9u3oRok0j2YfPvnt09NiNt/K4zdR94oCXuRA4ZxMqnNL+NoK
afZ40gp1DIA2RwHXez3EyVc1dxWrsvU+nXKfA9FnmzNGoCbnUMWQ4TCLZuG2
UqAxFwPA6iPpCewt6QHxfe0PgTUogPj6/B68wPo9oZkgFAwnDNk622WqCrgg
mHJmFbR5S+FMk6aboUC/G13TftYZl247KzYXoDdRFU1O0Iyg65cBMpQoGlKh
Pc10Y7QZ4gb3zJDBjlM3mcswuBjVr3QcpAsiDmV9T4esprBGMSV8EmmZlx/L
9t/TFdFSpZaUJactRx0cHRVHxu1sFEjF3TjQNLtmyt+FC4oM8hOU3OYoxY4Z
QtN6xF96gPh1eH4f3F5a3sYz9I4ba0pWYeYyxaViH/02UXvVaJEF2uG9IvjT
mLfLGriFxXD3NAjC79L6PTt3froc7qTyypWCycuXtAMqRPTVQEQYZqaihGzT
7T1ou/sWaZ1CTnu+eiCVtyIBlYMiOPzUm2/beygZg2HiF7ED4LXej+FM/TSi
uOahXv3KoK5bxxUC8pxMaTcrxxN7FO2CsRRFhst0ZkWSP2QfV2flbwQcwHRs
AgSVFKW5QLRTZlBDEeYeo/gdnmsldp+XJuSlkSiCBsExCfm/SU3WmrJDTtVx
ZpBR+lMN2m0TjeL6vUP5TAUmFPk63MGrINOev2AHohp5obNtT5HI27GCR+Dl
S2lICY7rXv2zW3g4KAndftHVAyIgGIca6Ug2rzGqV3Vyx3/ayeKL+r8wEv8q
HxTp5kAZnqwTZjiusWunnlF9B6+D8u2jlxWIjyE4dyoMmhByCUuwzUX2FXCO
VTmdM2CxUk1paIjAvVESTahE3YxkiJQcCUoHvaQXIPcnvnSpTXHJxFaAmQoN
b+Bdpuxfy8UZ7cZ69ERSIlbzPcAgM2kxFxBtJd/u9m+NvtdD+F08/0F+v0bO
j+vSBo+kaIOIzJhmPhv7p7eYID9OMFRyIwhJB9EupGTHoERECQ5wjtcf6Hhn
k4fZ3EMVl9/x9LlX9kH+N+umBz3fReTXpWOUeENu6fdKudlnB0N16fh+xX2Q
3D72T3XCwbZ753/OmV6wkm3V95u/L4lguWSHPQSRZfC3U96zzXxHeY+GYJeu
mHttn74SnUJMS7FyOizTIpZ6xyjryyLHng5eDzxMZFkSMLiyYymkRma0K8Xo
9s0zxjy/3JxFcoHuC33nS1inxPfneclnFnu4+K0g7YGsQ978Lzga8/03BTrh
05V7zq9VCXGO15fMDkROEvmaxIxKsNqr8N99bLlEZP6anU8NmGqcQoMnsahy
oW1e+JGTiTNN7ZLfB9UrstLg1bYByY1DLFNcI5o6P1FjyYc0DUtQUh8Dc+7w
jnl/Ttt76Hj7LjLu9Am+WTx14kwZT3kXcMr7Hf6Vfj33t9z9c13+V5k1chmJ
YFfm99SJ+ZhvBbByZNCc9rY/CBiw2WuUpDC3cORWOufBRZrLrtgSRLbjDFv3
btYMZua8er47j3oeVPXmey8ZvTtYuU8b35tbC5qMlF1dIr5fNfVTWjl/bUss
uRz3Gw4ST5FVjospbA2zEvg8+K+6GO7TIbq6LgNIjks0WojTHjC8IMDQu/QZ
vdjdtDng7Ss/ZEXhnJua6OLozdbn+pSqMRHa/NqCZxi3JnijvAikLS6v7A+b
hmr3FbX3PI8kKY2qNcSBvZ+3UtqjEp9ZHzjCzJ9VITwGZ9p9oZh2GeaW0swt
3m7cdZ3nCHwdNZjPt2+W+KKqa4yk938aZZ3i0y6FjWs7LjKM7OdbEl1BEKE5
aNqytw2REYkME+7K5n5fHK8UftRCi8NWbDMEhZyAy9Mj64yXaM0PTLONfjr7
a5k75d032OwvkMB2xll84fuu5PJ5GQ5g+DDGo+3n4HyfRojFvq3v7OJqG1uj
2rW9xwTj7VPnYZbdNr301x8uvLG/frnbiYp54Py72KkhDUoZLPV9+EZ2UO84
wkYZTyc2Y+d4V/s1XlJgQvsgp4qVTYdreZkkcxY4U1t5ESX72+Lcv6QLixhd
LTSufHrvSoydmbNBuQyZ2JkfX37dFsXdEfZ/UXP0EWq+qdfVeuee9O0pO5+X
eW09fGIj1eLHzypPB1TCLWtLLvBY8zW5zTD3crpc7Kmm0Xs0gmBtDOHjZ2Cr
DJFaU4bswVt8MYw3gj3lIZ8dhjWMNNJmlfjQaa7jRd7s9sr6pNoNfMp/bOuy
9OoTuM5PWzHO8vG0/mZOn9VPNMntc3GttpLTa/ly+Pj5ttyKaP45lP5Rvg8x
OeV61dxcvUDhZGeJ5Oo13CGy446fexOy69QF5uEjdS0yqNNIbvNXvTRlF5P+
mP2yDMbnth1t+9AkJaLHdM/Lk88XtHRFpv7frv668p7NPWMQxGDdIRvfq1c6
Ze+OBWDV4cT2shB5pvE5TfXeqWkzxvHaWaB/7GHZd62ZYqzHkBj5d9WZIh89
vRumsSX4NviZ0+arj8W7cFvki2LjI5t6Sx/PWFsj2a/FUvJ0SMuDDUox3zjd
q65dgpaaGUnPlBja4mzWkbFvNNTZVs5ekDcM1+FslD0NLa1PaDBsuiJplSlD
0jWS1YGpMQLps9zl1k20EeXyvOY29GyWxPwTo+qtnseW5sMzuYqFr+ohY8w5
1oydTm6wvXTE8xFuP86YrnbBp1vsjsZePbs4vR75pcjpF1dZZJBkyfrZhEku
SMtCNx1OO68mtQFDrKufitJ/DEblGk87tdqmS1+ts3kLPklWxFxlbf53hXuS
R9pi55Q1Yx9a9P2geUqSQl6SmYJPjWj3be97QXMzUVnK4KTd2ss9Ty81R5lr
mMtPu6G2WNJ/Hnx816brjcIY+JoyVzV7czJfwbKbvEJBE2p7DfzIHqCaS1LL
1hcR7fT09hLHEcShPjD17MKBbB+aJbHGZ66bZI6lKIMluaww/Davuxwf2bRb
4tUSk7Ur3yo3E5a5xaKVbm27nCluBV+eVW0Xtl2AhMUlIFPDh6x6Wh3udT6t
7aq1+PNOT0/aGI2LxnoVWsjzCjZJ8cu1ab7ZtZ9lzpadBHiGsAZtJefimmPn
k+oGT5pU2dlHjHMcEKGxWhjNJBCPQVgYCW5p9f6nJq26X7aQWr/EW7HQX6gi
fplAlxhhK66kJDS6B60y8IJNUxh+19IIzPREhMEGCZJr7Yvb0cmP19ctqGhI
9mG7plxJBzBu8WQHxHj4h2/WF3qV8fAXmywiVmj15u3jusNaCtKvphDrwd64
TiZZEGNBXDea1X43kKpEVqzEAEQIv/d+2EPdOUAPt8XbmnKTHmnqhPOKIqTm
PVj+Nf3vftet0TKpKMtRWq+TFlYEOyCpADPY+uCiSyUMhH2dnkmuca93BSjN
GbR7pmLadZgprD7/bYpjKSLvoc+KW8ZQ8ThyPdhXgiyuxslHyuHp2VlsX1Ks
qV0yO+dNqVGCrEBjBgkuVjL4iRy3lHlZE+9EZU2MGgkxiQaRHDBle8IoDDzk
ZtUYl5quaNEbMar2ZaqiLlKkKe48R1L2KI0923cLTZhzkoKuQ+x7hYOyvVTQ
qmxiYDWEyAZznJe7uuCiVcWTGTXFknSpUb2AsQcOZD0qasah1QKuWkuKjavU
1njlAsT2yGEF1Ybd/Xhg8hwcScqKJ44Ns9vZOu/eCdzq7CCwWQPrxqNvFlNM
j2PeQcE7Cog89t0TYvGOcimY7nFU8m0aDVwTUEBpv50GvT10ZdJGVCEzN0BR
hqZ89/NFHmUxIsELna4IBZhOBYkKKsMUSw+zGjOoTxijS1Nq6EzslBEORu9j
T2VNIPGop4evwdNnjCmztSFQrNla4lZWLRBVZlqibPp7gJAnk9S969bxNdc3
QwBcoD19Vc2pBwRZG9o9nwxZMVGkKMtZnUDRH//pXQ8ogCWA+xbiCQyaKhA+
Hy1k3YBRuwXZhlAoxHYb37zRQzO2rSOhY69pJ193cQR7enbNahUkUBQVGZPX
rZdKxJSkLbzjf0T3LtMGSd3WSozV0q68lSt3TruacNVB4ufOt9BU3iFGBznh
zScI2qIIYweDDBQ6O2FBWeZJ1mwwM/P1OI9SrJ2GbvhapQH90hAUdT8clUC5
oqc2kgiVFyK+MBvU7GuF6IYyoXk3J/JXe7c9/aZFZnV27JH0lINiUu28IZgg
uMUdKl6AN0Pvt9BRx9meWtsZsOk1Y0IYreWF9614Ol1vFVp5zpQM7TT8lQuU
ChIaFDRtiYNCM06AJEjaBrsnp54linnvRk5SjkHKTNP0tmab2lufMk8UJSFM
H2lHN45vaUfVh35hu9T70CibUAVIxJ5sKs6Sna4BA/DJoYsUICd2Hv9XOsqM
9QcfZF8mpBYqsjOSVptSsRyoWR3wuHdDDGLNRtswGQxhPnKZrWQhkEmFlIlZ
ZpKwbSpVaAkUp98yQ2aGikRPyd0hhqe2EoejUHamwzAkFERaMkRlhZLVM2zQ
CDlsIJr0wpiIslcyV6un1I6yNGMxlng+jKSdZctVIdXln22qA91ssNqFAOnf
F4PAsoORX6aU1jnokUbfNNDENHKJRiWPswCsy4CZKJ53hB9GuBhB1rH5XFkF
OWWXPDh/rDX3Xzshiy9KG4mnbhg1OHSd9ltUT0pIg5wPyxia9uba28mttmEX
cPTfaiYnN7cRiu8YSp8sOCOeeoA2ooFShTAYka/SPrPtQL1xdn+q+CETHrLu
AxQP5FdnNAYzz6MJv23CDEDAGAcUIwBMG3FKSvFCg93kcBKu5/Iw/E2Nnt2Y
nYZ5wzK4220+rPbCbR217oyg30rNiFbCxO1Zzo3ejotwv/OM0BImaorF+qR3
WVqOTO4YqBYlH5yEoByHgZJmtnDOu/m6BAnbjrN+7ZKNtUrbbBqiVBQtpG2I
jaCWqTg8E90WD73qgXECHIXJBSlvrmkpLlNrDabWu9/QmQd/boTHpSUYCCPa
yViOu0eXo1WDNNPFbKxiAyFazayrhbao2MFBjWilS5cwW5ZUGB/A8fPQnsDO
dYRZpL5QdbZ0jVDBBAEs15gzHUsMcwwIY/5dGTsuWAnkQh3zeP0+82AjQ6Nt
SVh63rTOXGz5vw/8JP0iCsVgKEFURUWSLBZEQBSCkVVVGRYqJFFirBYiLFWA
KKjJFVSMYKAoxBVWCjFEVSLJJERVkGDFWRioCwJFARjJEYixYRUYiRZIKQiy
QFCCgKCkWEUESKIqEUFixFYsgpFiwBYsFEYqxFjAUAFFAFijFSKQGMJFFRUi
ohEGSRYIIiLFARgqrIRQWEBZCLBVVRRYKCqooAKCkJFBRURUUVCAsBYKAsgp
2axQRVIiwFFVSCwVGRjIjFiwFBWCSLEQFAWRRVBUVFBQiIIgLEWICkWChILB
QiIQWEiwUAUkkRBVUWYklYAoLEiigkURGKgosgooCkBZBRYIiihERZBSKKLI
ooLIqkVRVBYjCRVEYAsikFFCLIiIkUBQUBFGKsiMCAoSKKpEVkBQFJFkhFki
oiiogiKCkUFFkUUkFgsFkEQViosCMZBVhIsUIsSLBYwFAjFUVIpCIhFIsiyC
wixQRkUCKKApFkIsCCwiJBYiMQiIMWKIApKHlkJ7ICyg+j2837ox6oeG977p
qK7yMOp5Lx/Jo0dym0+wFwIBiSxQq7VMumc/yoKGkGthXTHo4pp5yh848kHj
zKjnfw0UHshIhcHrB0fI2IfKEA7Zx069KVjvSeHSaynoJnjZu9RaL4IArMRU
kZHEEuVSJyyipWBBxJsb4jYol6aN2pf3vBchmjlEtaskRPxe6IctaUuGVxht
sOBHHZg9YwD9rAc80QIJTtbFhfm2oRYuK1uDfANdjAEZVQ7hVaNnC0J4bZg/
Go2jc4UU05VQLNRdx+EZxk71vMXcMuxUSVy9a/2egHqQN2C/q1r25de79c+N
/Veuoq/HSgvXtJ5G1zyfyX9kZPki5A0gNwoQwqJfE85ivq8ep4oIXvqMjNlY
lumqnz1UVrVGOo1vtvG54NFHttne2TWLsCbvCSRQ3jWnrWtrVk3vRxY08Lc/
sHCIL8fIffYOrItERXLPjlEEmjWB9YVC97UtqzIJ++lLA118vozPkqhaccDr
l93Xe9Y8ZuyyKikOhNu4RBMQwZZNaEVuXG64875VgMiMhoRYYLU1RpALWiSI
Cp66tdlnWB2iZlrIt5+e/HGMbvhEN54kTsopQ+mCBmLCIGvSqehJFEEYogKq
HSfp0fo26G2DixlGQVpE15VceU0y7ttIXAth3U41NdiQh5RnBhYpiUqNJ/hx
Sk4XeoG2j3ibHpQMU43zKxIqcOyd9UnRMGQqYzTOqRZ4iJEmBlmA/eSBiV8y
Ybd/fBlr64My+JqHCvDoib7JdGNsiuLpXHS+ib5GcqtnSSu6FyiB8vDYPaqS
4vlnGvCq6hF85wkEylIqPx0rVPJuyy4O7VU1HaFDO5jeTv1AVsEHzamEK8vj
WsKA28frrSfbaFkSD9b9NB67WFWo5gMMnWXIpY2XxfbW1isCxLBktKWG+riw
mt8dZomgVYOIUGCdhzeGsQQcvO9NKJHtdlE5eTfM1zYcLBkUeCqINqqnYIgt
Xz0hOrbMA+jrHSvQ5blapXb6zc7aWDW7mgsNZ7oN0QSHyAJa94rp5TH5Yxgh
HgHSHMqHWaUJpGQNrIBokY2J8nompXQOdfJnkXjpam52xjzEJzFrtHEVEqAS
Gu43pW3DWuXqSI4lXS+s4ZLyGGQnm9tmVREXylXmUVXm+ETis1BjimvEvUGt
bSobtdoi1orN7w0ylKOGOSBs6somjlFNEeLqMIMpdXyZq41JnTAHCp2ceiwF
4rAJNkCUoDnZNkAREuy8lT1Pw+o1uJfCw6yPA4CSgcSnttkxf4ciPtI/DXO2
s7ZsnbsQKidNF7hs6uZqlA9CTQ0YyaYRKgHmMxYMBA0aCoxmQwZHS2zJQ4h/
LNMo6C0R4rNDNpBaNOmnJ7pAxJaBv4ZEx2IjlJHUEBtUmTMuJTZEzSgkRn+d
7P5L5gCCGKIssSUxasbnyyLGloVBVKkcIMpkZWjiEFcK1+2l/il8s5O4SMPF
w9q5S7POciRRREm0IKQEDq5R30C5byiLyi7V1DE75Vsy9wXKNo9HsZ4opyTy
QKa5wdEQVZQEBvS5M1ttNY6aGjE2C0yFmo3F1OMQMrS9NbKjzulAc48Lsips
eNN2dbjfnS8hFKlwg0MyaZKjryQhsTapvUw138Ro4Fjt3ch7w4JW1+g9N+SP
FMQJFvZe9ZHDpXzHFIfO8V5759AWsocFxWSaEkwE6KUQBTtyrsUyekTNB9iL
Ht2xjedKtacxGhGbSnM476oTQQBJowoN3jCO8ZgMyUoMB1cqAqL270oInAot
mCpbEoiAWvQHwvcb+W528Am4mKeWsyQY7ekkW1fG+1iNjb6XNhdUvPMhMhiD
ieIgonRyJ8y3NLnJrbb46w5YFfREb2Ln1XVxPy1+eGOuNpdAIhRtMISMtVVn
T9v59vNLPBhsaTygPR7OqbZIGRXyyiEWCJbEpQZPKY4okY0sub0wFiRWMylg
CYNMsp5u3SGpPvJ00nUIm1dZOVHrj33hNRRbgLsayroC7nhyhQcKfQ2+6QHO
KCAyIqB3/pAFeDZoR0DjXnxC5Bmrg54J+mGk9TurptSlinowmZRBIEbXghc0
nGIRDVwFxBa4IgawoGcLNLP+N4uHaiLtYNcihz2qoYohqr1s8YMOHnRrK3hF
GlCvFJYYIOb2saXjGKUpp+q+l530O6uFfqd5LPx6/p0b2xqtBbM/ZnlykxoF
g+HhCHhBQFCRZCZ1vdOTi6URvy1uTPXkHthZAQIQXSlzES3kiUtHZKrwBV2P
mqoqm5QSHdCToQIonwyCeeAO2DYFLqD0xaJasslP68QVWcGwwibEhJYjaeDV
VwG2KsniblM/i+Dt8vTcnR+WQUn4En9vr1K85ExDgYHfeomxA2LNpNorbnY1
mHZW5Yq6gQOWS56P4zdCb0rbKMZTlbGV88oZXt+zZ8PZ17pl4hMwy/i0HNFV
zbWneclaWNoa6g+XQ5UpyNEPZJPQ+PYvIHQIcsOTyWbWFm0g7tIp5tIZlyZE
oPz9PYj6L9392WaYXcOjL1pUrAD1YQ8/Yvk88Yerr5TTzzvkUOCSJp+Ltesu
4Uiiq/3hvEKJIvABVUPX4E+x2OIuKBKuh9lpRRsOMYzZj0ThrpuPOesCv7d5
+Xyz9HbI9n+GkLs03gtYdwiUu40swaUUVFFBjFFhtAndMkBgmULlRp8yw1Gk
qxLHNKgZEVmhuQGaAvFIdGG/rVNI2PwkN0Q9bubz7ZObT9hziNZjV1rLop5P
CjrHkuyo6wR0i1FQ4RVR4zBESEFJIREDESoIkglZArAAEZISKEGIQANJArBU
LiqlRSRRzVCQgjGKDcQfWKA3EQW5pHSIZgKmYEJGMgIkgHIgVCQWQIsACdEC
ViyChCAiEFWABshIpAhCoIF8o/s53tufuZEvqhKZYcOw/MFPFCskKBlJJHMB
xIRg5ZHehMjgghJedMEiMZ+tO1NpIy+9LMJOoCCvv/DDWiVJEEOp6FsMIpQd
fx2RMqvqU+WcAkyP1snSWTENUIH4ynOtsB1gvENHo90lupkAUZAPTOuoL6OY
MjIR4QLgEyDUAg3YiBwyKKVAFCerUIs4tJDXiBn8UMpZWjikDiiPE33qjR/c
YYbLZnMZI98Tf1NwaSFYelI8zAHI8sjJqIhT362oq8xDk95kcRz230tja4cZ
kZjdFPviv10jNsKOGYmQMeBvZqnKHO+vUwNTtTkc2b0zJhqU+ARgSXdNeTYM
ExKw3TIy2htonSz2Yeh2kDfMGHGeCPBB0aSzpLehVPIiOdba1sAAY5MrLHYR
xGVmyNI2etRjcQApTRCa0t2GCUYL286IrhDuZ3W+QFKQKxEFGduTV5oZMwaz
375Tn4Dd6RsxEBkIQoF27QUTYDY2JtjwDEjCsEx5rh0MQKiXG9Rggq1iMUCA
a4fXbPt27Tar70yClhKF6bgoNkpmoQXJeYuYoGjfhiMcLGQUhOVJheckkkCG
MEkgQTof5qh7DlnrPhFwoHVCX57mhSkA4h6+EVx2LFDgzM6CQrbZHPp7HORq
btpg0KqGTVcER6tlW/G+79p4fsSLXKOZU5eW2XvKJKAaKBJEqgQUBHRvOH24
cPVZmKU3qkcK+bcJA409pRI5A3TFGSziozwxI0aOt48S7koUQQhNHaIzEck8
te+2xW8Nj1x+3859q4x6eM8/NNp4u/fprTIam4mrZFJFjEVgirmDD6265V52
qJyTv00HIcb/H3cTv65AOaTPWMjWIqWZbRtEv5zsVO9atyKRymKGFaUl+NXH
x1oHmxHG9IfcjRmmiKMjgapMCCtcjU9ViBQznRyaOB+WLzbuiCF5p1IEYXMJ
XfY7QXViHsg0a3xdrIEFzI4rVFFQfI0koH6FsEbWQQYYl3oRReczxu123Gmc
WmsuFx7VrfQyEdH5swtQ1mfxKVF0eyjptiYxski8BgyM97lC3tTIEWxvg8RN
g0hr2JWMDYmqJgQydtF9Ozy1hXvTSyPUGFZlwt2U4t25zi6Kkt4NjibHR9ep
eqDtTY02yh5pTGRbEp2uaprrqmWwp4y5O+XTd97CXsAqot1ADh1A2szksbwH
Brm1qB5MqAQMFEQEtwU0kpRq7FLTCcqTS2lvdLjnONuS57m07jAI9ELDQBlo
WYP8KM9giyN4Y2w56b3dxmHYMHUoCSF0Fv8LBY1dwJE5uEavxakvJj0PaZJH
74oi9BEIDVPX2SwqlGoz4+InxkrITv7zGrqwqDVKJw69TPzBG8QXsEy09MMn
TYuB/Gh2HZ6d9AQNaNMKkeKKoZDPGsHVYLNA3hikUwvg/MZl1hYihRU8yRg8
o4fiwLG7EX1n+3A2g0elQOLtViYbgXuDOiIPMCJwTyBMNUnph2zDy3zEPNpQ
qoScIMhB4xDpCDFrCMIMXJi2kEYbB+ugUrDgO8hw9DxHb4E7LcVKr4hwLp9W
zTFOsh0fCcOIuciehQZcZHRjb6EWPEUMq79ryVhEGdJkXAazmX95IrVit6vf
fDnRZOUmejNMyzMlQddtrYHybccJCN0fMpgNbMeMazOMI7McW+jKv3tU3cI7
GfgV62Pq+2uRVlHOhQib92NcND6gVW+cV+KwGrAhhDbNDRhBykVCPL+r/rn2
JIpbo+KRWTpyo8sDnt25i8w2yIxw6w+450mJNQJU5g0hAw2m125zM0X2eN6Z
s6jmxdGA4aTJqmi6hPXZouxm3qaO3gs1R147j1sx3mooCoO7MFVXW4Iycd1b
dRr2CpkZqzasNtQLMkBPRiK1IJN9840dDKz2YfNMle7YCZWBKED4SrmJChIM
mgNKHzv3SVt/mTRrvg+L9k6r6elPj9Mym3SQPdh7p+7izEFdUB73GRxSyhyi
5UOep+jKRwjH38252qR2S2/sgO/4iE8eItl4MW+1qfPzJVJrwWE4DyxmGvQP
i8HuOz4KoSAkGH5i7W51dhGwTg+S2JzkI0aSjmQOUJm7ASJYLPHfTLqaDNK2
DisM2zhYWdnuXp9KUdTbgDlnZUi5dah0fv1sM1dDal4Z7x5Rjz6HpY4iBYYH
LomBc2u7wporkIyi1H9DsqrBUkFsTxxQ9kOWKEFFkFBQfH6ZAnWGjCjN+Eu9
p4ZGXvG62XbwpJtlY4ay6BB2vTZ9GGR5oeidNLOkqIFe9r8N0XGT24PafBdh
1Ab3FgJeeWvfvcqzlqhyYiZOdNg01RCDcV1JKjcnZyGS4iWH7jgePfiaNfT9
Od6ZxCLZaSjkT8MN2zmjRBKiH7qzVEfBON6BpfYuENMVkimIN0tQ+3Y3Zy+I
sXnBfb2OjnvEXZNHor0HCPf37VNWgPgvOXMwVpM5G5FLkJc1uHGpGZvclsY8
rohTlptBS7yZ6xVEizGWHbEN+B1wQK+37yIBVzhYa2+wUDuvxAVQDvAqDVfY
CET4jtIQMrPyeexE+fFBstG+d6x6fniL1NaELSmJAyrAbrikhe0GdwxUXKJC
gaAtyLuUEpsoWtxF3uXg6YS69wsNQ4N/kJveT5ceE8jlpVdut/HW7KO9o5T0
M4cNIt2WqNS311hn1ZJXHwzbVDSb4JcCYc7V4n5bKwFVZDtlvWncTowWa8R7
9Z5oe2y2GlIwyKNqoYQMKULRVh44LNCZho2LGVvc23Zk3s3oXVlGAsEYhaVk
xkKwSJDEZSiJW2woNEiyXVLlEUQsRGVimt9dLVlSEzW5ki7JlLx4OdL8zelg
DbiIa6rosgZ+1M5zrZ0gADgYioxFLRLEmhyICxYLjiRFSBiqDCdNQydcyKoq
lomIkVkAEjRDkRqJRis5pYIgYGWqF2ecNxhDNbLvma/J39u2ToM343mF5byk
0wqRJt4zQzJCHDodudemjshBTbWTeAbRsnjO+6kDLE7xtFcYBhgzIR4dnnUi
bQtqEtCuCAsQFo/d9tg17cTIY2lU4QxJvokLll5lDYNaC0zDgIXlkhiFEDFh
nRZYlLju+Ty8NtzkvQ5KGcpfa1dsJnRNCHtdSbcWBtwp0pibR3ENrqmccJNt
a2mY4iFPuasJiqH7BB+3Xq/bypI61n6ONiKEVn0G1Z4WPwNaIEmGV2e9k2R9
IgW7p5ih9Yu92xq2dpdID1r85lk2OiCYkF9teY0jULelpplT1ECrCrdQooij
EdjVhBNtBsGSH6D1KAs61sskH6iMqGQTItue5KHvtlq19lerYqbm6Y2JtloS
6iJ1qk+jMMmttU42zKGuR5YECVkfqyzbOaGfHmy3Q26QQk+GdoWnerZZyRYR
QHPYKEAMSSyKhEUI4d6DHMj1xWwer9RciZSWULJkTAEMHHcONRrnvrHNRXW6
sQACSSMtuRKTTapSiBbA0loNsD2VigcawazkSdTGTDJobrxoPsY6Ls5y6HtQ
OmGQdliDZqcZmCswU6gOGjvksmcdpqOc6ffea8Mq0g7gaagTnoSk0B14Ip1Q
9kWRVLQ0BAcoRHiEaMAflduzybcvaCpb79+6Le0tWE1MKyCBMTOUiT1nh51G
/oFq5yV4alZEor3q6oM+w4C60MmOCrzOqh3u8y28sCHr0yDyDiz2c7oPjne/
aGhUC5vFVvlw3uoF8nuxJdtdviUlRf0cJbRWZhkMHjVAwfu1di00hYnfHI5d
a1qqmmkolLGrxtqHg1QDNi73TMoNYMam7MmJZPRBSxAHOO5GBKMqcvG4eVbS
ko2LdsdGsXjZsQUopxc0mKzIKAIfho4X4+l7xu0sgdcEL5jljdScTBpzsxsn
1mWxxmRWFIzD1FaL+53raCUBC7XwwiN76oJEVGxobM6vbZBAFAmLsizI4j06
T5NU3h6MA+/AD/d09qpeykhxRPN9xIBgJDuwDbwSyQKlkptqdfX7O6uw+Pac
UlkvvpAiLr95hn1P6Ry1Awx9dJD97R863JSDDQsVgKvKriXTvHWqgLsAOT1L
vlc8UuClp+ZkUDJ+fz95bPM2Q9iIG3iQp9/NV6DSnamkwP3RlM7aSU72aIZm
woycd77sgegZILuVpW1lnra92NmnZE+Otl42mfqMDs8mjpZVaHahjMZS0Bai
1pqT2Qs6nKivSXRrqn78+IqR5imcK+7hUtAZCsQgi0BnfxMVDr6Uo0ZYghzE
99DEEnFvuxTpkSWKDbHEIjbaoJVoyHL/aSc+wvcsmke9hsCrRkM2vcR6qWLd
hJ3+u4mQxJONucOTkvu0Qh2GbiLpQfl87HExo4Fg5nXmvMv+NqFqeUuxNH2c
4CHoSqXTTJkIKnUzZh2yQknUpgGYfALj7MSJAMwV0MFo4Xm8zG021zHE/M5b
JYrQlBVlTidB0A++2afF1FWpQpAcNFU0+LaWbeSLkbSSteBvcZdnsPNk1d8s
8r0uIPqkn98rmT5n6/Fr0vNyGvztpPkq0nEBGGfaUY6rKwxfpkVRBi6oN6ov
coSnQllYMz+fyx3p0PsHuKWYC73qvSodmYEvAVFZQbr1+ChOuRLJSrEsj4jM
Qx+bNJpHU8OXpZpkNhG9MhczO+eUJMOsLd1QOY2sAAwTRwgt2rzv8U+RObZA
xaR1dTOodDvClJm+MzpvLAuzRfTLt8mgFo21KtQ4C1GCx1CG7Camtz5eF9bv
Dd6tMQvDMSESwOIhcJmjqMbZ3uYTwPHZtrjOiUaQWRqgWIkFcaIpU4Yl22sj
u79jWIazoWcrSMYDTpU7WDiWXfUNHJFoTJajLtbp91qAUQQvYICeApdsYHLB
xcBpmSxJkiMdEAa2rlenEUy+EZ7iCxazXAjPd3MtWQqRUVPVnLmzw7m21wb0
lJxggbhuAwbadFUaKOrB2SyMFMr3r3tbTPdYLgh9yEiLQi5R/uftit/x+ivS
tYThYKT4rBrFWI8Km3zCCmAOe/ybfpccMp2abSLP2vDkTYe+Jzxp3KWYfbYz
BQSGOJtBgeTJc1CVkhMJMUrGFZRKyVMhb+kNSAayXUCJW8MzEpkvwmMRHUsn
om3Q99jqbFY7/RNMnFOJGhXU9/trELgZd8TVUeW+KJsXcyvm+MD3TEP0BDym
yKE9UJswkB92pNZACDlPe2u8z8uQr91GGXkpHiO0RyZxi1SeeaZn5N7xYO+c
DEcCnHjQ2BRL9yHLBOEYEXAmPDUJ+/xUjuopG+cI0g3LEV327bZPfMP55cu4
vYbjsM795BfPEfF+Cw7KqDOF0y1BgnEbPY0U6MV7rKDFmZyjVH3LVfDdLiJI
7CbOoBVdrcVe1iVFzvs+Jgbb+TtUqF/LjV7jBGMFH3doWIf0M4U0ha7xF9mZ
1UHFFHG+RMYx+jUJi1nfy9tBVFIvz5e2lmrX0vO+eNmrRIi6wlFW9gyr6gJ8
ECRUKOnizdPuiwDynyzsnIQN04Cc6sfhod0gaZMigvq2EigYkEQAiCRRAmxI
KNHBpPabLsEQln1zFKdkzR6pnLPSFMJOBFenCpImhj4lTD4P+odtmsEkFCbW
oMDZybnH4Joe2vHzJQzZm7oAyKrATjmzh9qeSnu4zB3N2hHoeLVCRuBMSuHH
43+jaoQDwh07KSwklSCdj3znVEM6+OiYbWbR57mSLghmvIbXynRnHctghyqw
wlknoia1iOQbCjPjI0FY0LE/MIKP4Zh1HwZ8QmDdGaBU9xoVWjZBzHXtAaCD
rB4RHuirjodnWWwsgSiFds3mFTpHyhel/L7vm0PIN6XpD19aPngHcQIRBGHM
WN4+ZE0HlhB26hV+HUEBII19nTdaVIon5iPRQ6TC1YXu1wvhEIDLNQvfTY4p
pDhGsQ579mXb8ddG57aL86zgU5CYYlpMxSs4wuxpEO6ll4LUoGbp1mREo0AD
KKEVEYsU62VDsMyxfdq02BJkQ6D+rEF3dg08o4kLZaDz+aVC7BcNJWLEKLay
1uQREFcQHveEvK+YL0u5BjbjfbTKqRsxUtAMnB2NIA4ZGgvZu75JWOoCIG5e
9LiAp2IUiaBVgQ0OaKQI3pZ61VvLoB2Yi7qmjLOTN8YgkihaSAVBmmWFUqCs
VmVM77SO2k3dcQUwjcplnYDDqAqwHQcykxEC1XRUkc4lSb2AZzW+wepkCMwx
42kXt/5T+7AX7e3PT2mN/M8awVIe1MeK9Xu5d5z9T17hll+fQWPKMvW0pRZD
Rax7/2dijWuX1kyRzf9InU0cwgjRH8GWYddSoedIbXOfnR6Hg+jFVDeaO0UN
V2AIlhJ62qMzWfFLx2r6LlLYyZdl/RCOxMGLMHgVZW76mgrd2hfnSJq1eGA+
lJYiZGlGxzC4XXVgHEZakmptJuGK9nkJaiwUIGAxj8VnECGcNw9Uf2woxRVa
a5jv2cekyyfYw61ZTbbthtp0wQMJkUoIIrUiUPJwJJndyUQDDCXjUQwm+qBt
CVkmIVCLIIwr8eDfjahzco+NxhMArD2WaFg6HZFXq9OeVNQfsbQAmdD9e1fT
6W1RNaqroekmdunbzydUZKjya2efLWsAFQyRS6i+3uqx42YWNDh2CgK5cMJD
zSFu1JStVChpIOVsYF1pffAVQ9eLXsuQkMmts1MEPPzgaN/xSqP5OJnwZyt7
kNbsuK5Rvo06yurHHTQEsFdszxRMOAgtcMTI+WvDQ31eM1rrJfXGjZeYYm0j
CiBbrqTeIcN2ONpIchk2E8GcJ8axWhjiuROp3CBXrzBlaBW2peXlhLehni5U
UQECA8nSrnCrG3Bt+bfT+YZYDrHWziORwAU2qnUieML6/XvndITnnuCkxgVV
GBVa0NiY03il9O1GN1f8rltNHIlnC2yR2N4++/aph+0Je+NHLAL9d6I9yKL3
iiRNnGThSuIwA5yAN3iZYq60pbwM+F/KlkLH9Vsl97ATHE4SwqDfzSuPLO9K
DpRClFB5mIGa5YKCSq744Mdk1MWHYt9RdMFAQMnLEUN6AFlRcGMiYVXIYQNA
2d8F9c+CifiEuw83XJm08p/Ua0b3sE/eYL9oKNCrSEu2Vd6FWGox1jw8MOml
q92SCD2u8WGxoccsFAhzhESpjO1BJiAK1O2NCGO2KzEtwuGobfSQng5xI0XX
NaaMh92p62MPQVK8IVVQzI+HyQ0LoKLvU7FiN6LyuLqjGs2FmAT8wVd8VcTO
m+yRQh2vlO7vSAwZRSYOWjorzNcbOQxvH5Bsoe2YaINaBBuEubLbInB4WZ6f
RaBxGnDhBZqNWhppcIc4IBEDY/FV7j22dUtdbOR4J2wuyzMkJFs0TqU2SsAD
ZgCi9RC+pZKMgMBkNulyLuh8Qqe54VmW58Bc0RviTdLCAghAeMOwd2d7VqpM
xYa/laa8vXqv/nysyYEFQCU5UMhLQkJ4qF3XivcuW7i69WVRLJaEwoRUQMhm
LC9sNgszjF2FlftJKwVCIYdy06+XTAHJi3YnMOwipyikgKHFiocIAPMigdgH
xAj3ndgSnAF+c1XbgxalKzGDoGIyuEF+3zEF2rsVcGPmogowL833GwMaFgDW
23AQtM0bTCLvli2plVI2MFefY9VxIc8Umh9ouw2CopNjQCgpLwtKILtAkReC
WUm0QD+L2pUxJMKEixB8MMhqRU5jPtNcr2JPszOGmU+X4a7jA8iTr3rCcWbs
kCewdJQztYdE6IdPPOIDErbvZPPNHe9S/EMZ0zFAncZuIZDqnDzIg+Ck8pkE
4aiwBQSIaHTgmDFhRhFiwpa+Aec0nIRogYlWtI7xC82PaKXWjB9ZM92ihozh
janqKauDWYh6sQm6JKjMpvftIvOCFVWxrEUxxAZneFXJULBw1XwyGs86j+b4
HTIYoFZYTI2qdmr+0hpbXLNPwzxKnzD84H0pX5IvW65J14CoutkP7efn8x5+
ZPNeL+y6zDXUBEEIvtlWpzrWDD6wjIb2k9WgJW+cnRWPuh5xLLiJu0ixl18P
yO4pxKCyYCaSWNfw9e93uq+uei5bUMyeE2OabqDlcG0pgPJo+sKcC+kTgYZK
4xyWevszkxtCnQNpV+vl45mdxDau2xsTSF8tI1eXXx6+NPr9P4uXrB8xHL/O
zgXvo3FvfaQlbzatWYmHJB0ZGYWAsg58iMcJZ+9V9oLfFJkhpRDgLSSwj6Rs
/DRN/j2mmUNYgkkaGdqQZ9Vk9M+bUD0yGGbr9PpxTI7lfeVwPsHEU97YZ69q
WW81RyJ6WgzDfBrybRaFFSauwLHDu91B1kUgaRCipcrIeUALi7aRKqQH2TEs
na+aw/totkkLeOcsN3IAwwYpUfhnAJYgVHVo8JWRkYzbxBtSrVaLa0i20ihU
6uZMvQmSEEusVWtx+a+Tsk0FyYkaZ+7DSYTYGbKvWMg/lWHj4MQ5bubx91f4
iPxP637iTmO6ZxoS/L3nzkHcqxlsLQlkcZKKhojQfLnh/TgnqAnVkCWkKLEM
yT+cD5HjjcZxUFHFuvGn7Wk3sfbHuMVKJZ6Z/tSjCjXkcMK/iBLR6pVmCTjg
Q2ds0ZwUDLtZ2iFxDW5VRplxJY0WIhAoYvYMvGUBKEgmnqoN9qPuj1VNFqdS
lp556t7XHe8cn0GzeM3FjsKDnpjWRxjCT1XekHqFmvW1necMHLgglSBkFEE2
iDMfVSMLjAmdsL9ue7I9lqUTONsiR1jztNnk6sQGIO+uVjQyAlPXJRjq1fVj
QyG4wYrTdKD7YuWN9arIKoFb3tR/eqOEsoN52kgotFRIfYMPEaLe0jqjjp1m
3O8TegQIAdmuz+/l2ePGEeDzlGb2O5uCEQSxWnQVPOEGRy7UUAIyB1sceVxQ
6b7XHSeimRillZfQtOhcZbjvnJSYBt3tC1vSaEPvt2Rpe4z9iFTARsIq48+t
qZcdzHGSWTCXwxaAZZfxScuK4o/JK+16AmPL3PYFUQN+nT+YjlLBSgShaL/Z
Ws6M5Fz0kXc+l5tbssmqohC4L0oAaBnEfHT9dQ0YC/HjbS+p795Pcw4MqtMm
arwHo2W1D5pzrZJXZ+jIamIzYkj4XwqU9xz1xJknxbDhZHntvsoWTiW4eCBB
RXBTwe0FXRCTy00bv7gGTM9NRYcA++jUoibMgEesvTEn9HxH1CDsLXhGkyu3
fizV6su9+24tPwH4xaoQ3NXk+HingsnU5fgjawyS8WupKzPTbYpwQzJiC43a
SKUgQ7kFBzc0V0wZ43w0TK+4CIKkJ+CogjSbxyy1omzRzWP80hgEIVzfrM+M
whpCMMQ22hsWbJy6M7ZAaWj4+DW1acb2+QPIsgWvSIqJbs89j6fGo9d75MUo
s1d38dXqIkpQoUgEJUT5+ecL01cvhFzls4+eEkXGFNUoTUrcIQURPXAIJc0F
h6LWEt3P11Mxqnx5V8X+Y9fsJQBI00ylHXzlPoMsJBHnBba4cNL3MR5ft9eJ
HRgnqUWOsK31AapTbRFaMuLH3HzEZDv3Rbrm6JifiKqBx0cpwfaqNeNCQ30y
fwPenS3EHWisUB6V4CAKLszczhUXr2ceWu9o+r2aQGTEC+L0pg9vtzi+53JJ
0EoBZZw5Xm8ZE/sOvEG2vdMBmtHzmja+2+AwKnW8kBGipRbNW0SdteqhT3En
6H7UIy2a0Zy0Nt9fDOxRT5arOcnxpRwNiqV4qkbLqXLMqkoqnXRmbNo8MzcY
UNnNTBnFhBQEEUHE1m2+b88UdHxJQV0UiGIqE6rVMlBgK4u6ZyQ/RERAiP+W
+VBvUeISjVPle47QrRI4G5E/OcnMEAgm49PvYfX3XwdIiUG695ZSMKQZt4Wl
76ZUhzK0enCbAIsEgSiVEwqhJAUm/1WDmwvI9WkbZQLN7Q1X6VrqzXjvzQKo
4zYjOwm+eAynnIAkUe+0Q9tTQmLGFmpojI8XpntjuyoIxPDK/trruhqQPxHr
Nuzd5qJH6MQU2g7365imz758ydvtAaNCM7Gvv0bbWqWwGHrtuJqH1q5TpZRm
tryZCxIiAUnmgUAvKDEWrU1AqA9vLVPscFatAZMFLASj4m3FTZiiegWVnaWX
nCyUREyd68AI12epLwQ9jdgLC4UzYBllH66auUhbZffd0RL2IZvOPhRkMXFY
zHUurHhVwLbhmlUwUjsLULY9BKW9/Pf4zgSl+LDNVoCHw7P1k1QXS6B/jQ5p
TbxVHhVriDgPUWlKvfTFcF5wwtgSQg9Rnveg9innwtQUynvfHMD4ZFnc8Pgr
epWbjZaFR58QLt7QS18ZGu9JxS/g9LzzHrAvT0fbx4z8qF9vxPEyRURiCsYI
kR9/dEz37J3Fu8dZpp34vcjnylk2Pk+dD1Xh7FETnien3wfPGYYht9ocZ7H3
q4oB26YbBdSQ89io4I9SjQN52aDx1tlEU28q/So1zlfg6csLInfg7E8oqhXY
a44pTJZzoUW3UWN+xHgNyssAEzRw89M4y/Sot37ap6TW/KcqlinB1RAlPUzm
n35jx7aHqdKEHv+3zrURZ5w3HbtAq6wCR2H6v4vJNF4p8SNpHaMIsBYIxRQg
66fF4c+4b6/NoH3YfaJIen4za1RRigimpC1YiCw6e5xA82UAJogAa3y0ZXSx
5ox5lgLKNtJdvWlavEbO6JYVfUHxob7AHkYNC3JWc6eqPUGZCC2viXaVfq0T
GSh10dSlOjixKXLMoh/RcSnwzU0gt8l7vWY2vsc2Rv3MY8RrZCKdoBsXanzr
m6ZX9bybcW0aPdJBalKATM1SIEjiO4swHmsi1Xrv0Eoxevfi8NYWhnl57ei3
CYWfvREU56IAoiun6pVl7eaBYUoQBBrE2NSz4WjC3BG10Xe3BfAjchFYBKkI
Rm/zoUKw9EIgVOs29DW+ngEcVPorApiTGigyyedVmomKIDwQNZ+5E9qrXPDB
bbyiVvNbK2p7uNBdvH3GBWF8GCASFF5gaKHglJe4uu6Gwhi53XH7ZDj4xUUM
XIylOBqEtdtNHCrVGCdFStqIi/jQYUNRhjlYFSOxJsy/eghaUNyyIQ6MBmEM
CqwKwUP3Q9AMh8qtNMJ8O4gyXn2/j2PjXhXIwA4kYBCKDB9fxPz80R4mzxUR
VD+BF7gEyC3TmYBqk3poCyymcLAnRcXv0HB0ncDqTA1JWjSdP0hCWT88x38Q
qU1zPeq9GnHtnlYevaD4ALh48+MdwO0BE9L/Ij6ClzIJHsNrvxzLEJmsvHs9
4uEgGlSOqAAeGdr/FaI8Y63O2dtXr4LVCjQjydTfXn9oF0hAPYOm+dUIwRAF
AbAo1AFTtsZ8V4rSTbjCnf397cV5djrvJHbfdYyXpeKPlGLaEhS00ohAa0vf
DohTgQf4UKWw6JOMkbdoedOa6FjmIrSPWIunGMu9ECE6PcvQomhBSg0PR0Ge
wiu0blSFKJbLNxxp0ZXCAn0Z9PYgCGeV+25lUavZF0pYFiUoEVr8dCrq0erI
9MPNypjKoJ5Hq6nE9s6ELYHzywVVw3ZBO2JAufBqDHrBOIRom+/pr2/Prses
I469UextO7TBsaYMbZjD8eJRnv6yjznhlxvsFWvaDHcklVXBi+xtMRIHnvmd
uYu1E48ivZ0qW2zmOl6nnFqEkB871cJ4xM6xt59psEQKpCjaXzbea8V5GJck
d9dAOx69nnndUjjnh+JIG3KlwpQgQ3NZyq1btKF1Q1gcRwkbWBA8vhmU96N0
YoZ2DhXJADJsTVcwRourFqkO62EW66r5vX3r8n21YtTI3bDhQqgBeFQguWIC
jC8ZjiQWSDV2uQWKpWTEAWhsliSCGmyrgdGjfXXkuq9eNjMv7xpo3e8uqkHZ
MTG+Vh8cWHf3UaAwCUxnUDtjxJogr2nhalKAVBhjTiCYnl91qxW01tmxJR4Y
DpNmgNQAzE1SMKQQg+ICggEG0huSBzf2+nEzBni+XrPx9fqO0O1tkx908Q+k
U2gtnvqFaMURBmtle0Ih6Zv6yRTNDAUNIpCMfXWp88MbcRECtuca0uo9/nSB
IPXTldgETQ5B5iA7MuhhZGwAuls88/d4/Pw8/r+oXXyfXH3D4eGe71/PL7jW
tgFkPtscLNqvP54zKSB00Nlyy+QMhsPWiGeSgf/EnlpeN+FRU1X5/Cv9LdpW
7hkz1FNF0HqYY+X+uKY/UJ8P5Id7TdwvlDKDgMvnWBagDCyI2hfNUQSMRz9y
981fokWmd80f3bUbY6INf+eNf+tDIv/XKuUFAZoxeCSE7xRkjHEK4sYzDred
6SLwyqWVjvKhg75jGPRwAHIeSaFQVg5MdTDi0bKPCAUJbWib9A4TnRIIIHFw
777PEJBX7mi/759LQyl2HGPOgK7/oxxxycoKaF4vfPPbNTxAfjg4DY1TmLVU
E/aSjVZ+l1hW11VQK/LSGSoIN0HPAvzYIU62dx54/KHLxcOO6VYaX9Y9Z/Xd
ECoPhX3SfvZNIsPeFte1vVfvffbLyfjL1fOCL0ihkyGzPBIZiSQlteNPf+qP
VCKNRN/ojVIGwgzMLz83/GgFQ3RtMhMQ2FIF+qYmn1u/TiqVZxSSTxMYwDbQ
pV1juYCTmSLluJpQkkRHVwOWYOHaw6qNxS3AkrVwvjvCQoNVah6MAPYSD9yg
6k615fUfOC6r58lX8fFB3TwghAnRRBygMD5D6O+E2VUXX6U+1R8PolSdd/Pl
JzoBPZxo7tVG/X7bYaV18T91rSpaqOXX27w8+/7H+Lx4/Xof0G9unS7bwmqs
RVAIFlqEgPRk/o6MJ7tlb7iHGQFV5c7DwyGeCbP3+Pbx/sAJvERUGeyG34ew
5ZCrmJGeYb9NE4PRrAHzI/qfNgEfKhPVasHZCKaVyzaAkTsZe+Z/N1RYMX+u
NAJOmhQr0EkhcNqXP30bFyDoQIyvdLIek99vwM+/29CYQfj7/hJh+71UoywE
BfJ5gT0QF/6TJQ7irriqRSuQoR+M4yA4NWOdv7fz/JFl+UgLvpr+p7fQSSEn
rwg/XuvSJX8fv9/cPo9o9a87f/h9v3uZ0pl/TC0n6NIP20hViBHUzODMANKC
+jzCottxNnlXklgikkUiukNIJDzfppXfzcdNIOfZYEvoY4swImSJINSh8GBb
DX7ab53/f8ptDYNgxtohJFt5Q2jyp9nxcg7DuCzASTyo0BXZF/mQKnVLde3d
hob979h1c/FzR+bLvUXctZGLSVMsrQIhwBJEMVMNMwI+3DFgFQzRhUxFBTm1
XHZ9/js5/FeH52pmEkhJdcXQyZ+r6O+0k7go+eQiqi9hPHFc+E1enfVWTFtH
J1YeB8aFt9IbgvoVUReChAFThAUA3NdThg31xP6z6oYmgqizSM0xvLxs5kAR
9XogsLdEeS1Dx2RVvTRGV9qPWmhti9EFdYM02letMsZIHlnNJNQuZ2MQoR/g
1wYs/4mvwk2P99n8vGof+9/qh+Arn9nwfP393cASAQZ94bC9v4YM/t+ipTnx
8r9evg97wkg+LR8VEvo0la1/LRPaNRF28oR4/RELh8sG8fZpvuHtBBBJHDA8
alnt3RA5/X1kH/4u5IpwoSGABpQU

--1171294248-886438723-1198775182=:5207
Content-Type: APPLICATION/octet-stream; name=kern.log.bz2
Content-Transfer-Encoding: BASE64
Content-Description:
Content-Disposition: attachment; filename=kern.log.bz2

QlpoOTFBWSZTWU7zf28A3nB/gH/97wB+////f+///r////BghT4OM942AZT0
aej3PV3vfK99OLuanLm7ZwgGgkUpwA+3R6BW9uglrOBQ0h4AF4K6PQXs8TRA
DueubN7zq9t57XcroOg1pvLKmtHuqKk9wfeavbN8o0PpSug77D58PL5vA6aX
vHO91s3Og6tXE97VVQChvceeyAAlQ5uLiqFPee9npaE24xo7d2dUFChu5e86
AF7eNYOh2NR7z3nm2AA6Aeg53uAAdAFb73hod9WyzFRresuy2e9vXo3wAAAA
J9mpUAe99NAAJHvVgaFOtNvQ3WNSzFNmbbVE2Lrcmw2jVuuCgKV28x1zIUUJ
X0yCR9Vq9tK2aJAutaa7Kwm2nbUEgklSyZS7Nu46SAKkJShdMJIgCAQEAAgE
A1TDE0BpqTJk9TJtJhNNPUyeUaDTTQEhEQhU/VM1NonqZGGkPUNBoAAAAAAA
A1NkRBIxG9VMRoBoGg0NqAAAAAA0aAaACU9IkECEamgTTTJDAI9Q0DTQDQAA
AABoAiSCE0EAgm1MTRpPRNkJ6KnlPRNNPaptR5E9Q0aGjTR6mgAVJCEyBNAE
0Amg0CaaDVPU8iep6Yp5TQHqGgNHqAaafOnx/sKXyqA2sf4z67hhhIE+3AAI
FRIC2H4WEs2ZSQhFKEUiBEEkCKBAFIojUAWhkQjGBUUoYNAqlBQpBSEZAYkY
QCEAgrCqRUr3fXNP/H2V9/3f9/+HxtbecC1NfYumKT/C9GXEc0VMNe10Efa5
aPuueb4ZMzMxkvrQw1ueROTjAxU0bPio8xn0SeuICRFWFtie5kREmx4KoZNB
MCntvWabdltgBgP7pRa01mYkdu+PObOp/lbISC3Om6SBJOGgTzZ23OOM1f2s
1tf/O69PINKYdbQk5i7O3EfhR54TNEm59Va5f8Y1/1Mf5H8Tj+70jDbK7Qr5
490JBEN3g2zOxjGM9gb8bhA1s7MYxjGbB+R36Zrv3Nvkv7XVB/x6zdmfzDr/
zVQLAu5GGog5kzEpJKiqpVFIJJTPgQqqSSqvdefwRu8Q5y5S5yZmUkpmZmUk
pmZmUkr9VVVSSVPJDD6/3yDZH4pxx3Aj1ZaW0zY9XOCTTjXoZmavHE34NHpe
UuVdfSX8ff7nHt9VVVVVQOfX08vPj2p6+t1XbD7E+Lr8bzzCqJtOHtqcrp1e
tSsXTKtqLbDMlWuOr1qVi6YayVa4w1kq1x2v7GpQYunV61KDF08X97UKxdPN
mIFrj9K99QDF072YlWuOWYlWuOX41KrF07X11CqxHTlmNFVa4w1kotcYZkot
ccsz6kKxdOrNahWLpu/GpQYunm3JWzWLphrUAtcd7M2hWLp1ZpKtce9+WoVi
6dXrUKxdOlMuGgmTSmXZoJk0plw0EyZUy4aCZMqZcNBMmlMuGgmTSmXDQTJl
TLhoJkyvUuVi6dXrUKxdO986gGLp1etQDF05e+oBBM+k0EBmmIqOU7BhVVyN
p2YVVVFU7MKqqiqdmFVVRVOzCqqoqnZhVUFsuzCZm4uqUznOLmIA4xht+GKQ
xjHC5iIOMY67zFKCZnyO6dmiqeoqnAFUfvVy4Zon9RNBAYRHoL9aXkifpSqn
oin1bLhmifj92BJDMJl9Oy8kTjj7S4aZm4unoip6Qnjs0EyXaCGclH6IQQLX
HLMaq1xyzPEKxdcd867wqs1x64aaDCZrI7p6Iqe0JLATM/MfWnAEEyeIQWbw
s0EyXEy7MiY9JoI2JQ9y4Ez1kCSAzeFhU1uNYA3vq0FzxYnFGBbAwYvZ+50x
MRZKITwxESmVElxLTKiS4lplRJcS09/se15fXf3+vQ6XaVmSw31UMcr7Xd1N
l7pqqJovVNVRNF6pqqKYzXL9Z69Yh8ecdZhX5UOl+Vd/q+j8nmVzeIQuLrMG
S3yg1ay6tuqxXp47bsTDC9LWe1rWLWtSvAuhxADFKwaABRaEpAVpKVUQRFRa
QQRGhQQaaEBWhaaqFUCB00gAMi0oiofxUBv2Tq35WS0km9eyvpVaLQotK0qC
0oIijSokWRZGSMKbN5H0wdxM9XhRVyuHDv8/d5PHXWv4o9Bc9jtbk511PlVg
jzg9wuc3mc6qoi+teq17ZIgmkO7uFN5xaZ2slfsjDNp8MSGsJL8ho1J8V7V8
GjXLdyO6vafJlBnD33IwTeR/aZK0T5WRAUN0GRNpd/djt9Te+ZXUfUx4I/vc
AZYVUVSlfdDxJU5cmyM1kZSiZkQ9fRnXdaPVXt8svd8p4dtZ0g9bDxdzyqIE
rjALv/0Y+rziAJcNhI3qm/iqxsYtI+giFImNEQ687SRDKMIwtagskD+01RwW
KCQNMKJVqYEA95JSCqqqqqpSqpSiUFBpEBcIffSFo1BP0D81DyjfKKIuYAWK
pFRoYrPMW/vuKZMXfaWsHphOr4+0iZylZ/DryD/MUHAPy12bALBu+TCBEqIz
6g+HA8Of+ojN/bWq080ei4+eEfEv6Q4MxSu46YM0sHm3x/L9tDr6PDOV+zN8
NqzxxebfU/eTPjk0TUMH1mBzAgQKghD6/Z6U7j+vjQnu8GdBE/u8ZiFhKdQ5
n6S4auWo2fDe6vl3RI9Vfz/IyI+8jER8tNarxvSp4pae76J81B/XxKA+k9T4
tXeGvJ+q73BN4rqb0UXPjyi7XNjJpfkuiUk5srkdLqsoCqMBAC8UkZcD51T8
YXLsIaVm2RuEfgDVuQ4LlG4VwIFD045BdhI0pS1KqoMgayVRGE+emr7Btcr7
/OvrPr9aJ5e4wf/q9n0qIAh6j0+Y9H/2Id/t85v7aPMQ5T+JUKoiBgNdj8we
ClQpCBQYPWfS/nfeWD3ZTdsj3DzP0HH9X6C/Rjln6KWJtzAbwIhAEHSnNWII
TMgIEk8QgJRmlFh+Ojcug3IXFVAbvBa4USFRJ9B5stsWqLUzGmuX9tsno+c3
Zsq+QKGf1GFlTrx9Z70X9G35fxcevfExLiKYw5dx13NsmQZdc+9m27cTBY8v
M+cfpywPE13Wc1zPU+Mn6W2333B2xBOv2dYtfB1FgXR7n5jgPs9Mz26M2NfA
23yIEM+FZZGS5nl6Kh2L9p42MSUjpA0lmvIBgNukD0CXHrETg2tGXW0IZbMg
wBF8zAJ6n79isz9G2hlm7sH2kEhFNGQZyE2PnbNFvVpX3AkrgFGtgGoKyKFb
RDXU/SEMAplICkXxIUrM+FhK1iF/uD0ZPMX69QCoISIOiRV/MFmmW2iHr+73
0/1/h8Pie75/JxD+3mFFig90D6Ip06oh+DjRwsgDDBGTIH5BuOz6vZ87N9Hu
8/Hv17EJC1Dw9+gc4LEXh0++xZ98EITNrErKKX4W3CP0VavrrV/2xgMkdrYf
K8WBhDCOEjqAjE9z8tHDvknO8Pv++tvo+2gbhhNw7vW5MkCEb2CrZqMsx+MT
cUQUgCPhT0hEhuIL5VBIowS5xm27xnMS+BJZnfI9LAihAi5Y469qGZN6HeBV
GzFWxP8N7z+wXcVCmJKMDTinUsF8/B/w2A6C26yZrtRN5MagHd2SiIKBsDwR
Zw8BSsIPeip7/TsY5GioO2Jl4r/4RFmvtnSpI3PXyRcdlGWHc3SlNEqopk+W
z0KQGjEHOY6D4ja6m4oR59x7MImgHSUXqkubSTd/89zM9MZmP53CcAw7iOwI
+Mjs9z9k+DEGAc7NA4ZYOHz8OGgXAVBF8OYig6ox5yejCY+ZzEENZon4m6Oh
3IoHbtu5eAOonByG0EAj1zEdQkU6fyX9mJIxOQ7smLZc9QCQEhCOjwHH90YR
eSAhm65ETX6zv9t+tcyAORUSQ81ISaokUAxUkiN4TI+nfdPPs95vERBb8O5P
BdvU5EntQ+zvInvlCSiwYfd0Y8ivgJYmUdeUN/rPn7joDbmWN2lwEsoy84k2
SEdrKpnT2BolikperndCLJW8mrofQfJQ9JlczKFRQMTDzt9C6fU7N24iiNvd
6eUq8jpSCd52vkMVXE7ateZ9BElDhARc9p2ev/1FB7P2fs/W339vGiPPV0j8
/T4ekSm3ZAEjiNxP4eRUyEpnydX4unHqErLy+cc7jRd8+M7Edr2qUpSlKUpS
lPu0ZqgzVAYYQwI0kjcR4Uj0v30/5MfLV20/gqwWXW/zs/UOZyAQ4uHL129U
Ixi+QH0lDxsnS4Jj/MIIPu3JdMKNvufr90ZfzSXISiQkkxfHpDWvlfu7zGkd
0u2NSZJckkkkatW0L2lpGrLJSyReiR0OtVgRwMg+3V3JEXTNXd3eGZGMkl2X
cntQJTSyEbkbSsMFgRrjmUYZdl7RFuXckiyRSXLd5tcdSXd3uhHhq14mLqmb
v2m1wZK0SqvrAjvq75YlWNBfGXRMy6zUy4skiDJq7uXuWW4yS5L1D7ZowANO
RhJJtfe5kkkkkk4k2k1q72kRuSSIySSSSSSSEkkuS5Ef29qDReWUohJqi5Rm
M3edihIX7hdCaEBg5qRCHsbddc13/fDxvsiCKhtHQ+k41s972I4QwmmXI6iH
WQ3Hrf1nrn3+LOIb/aR9EQ/Iod/U9W99D4b1fHrZc/2IsblE8X2+xeT1xDx6
WWrJfVHRl8WN502WWzNz0Ah3NJmGYCR2OHQR9f2xxGhzrt19Pc2/qtfrMvxd
m0LZ1FeIfQfOctrcmhAqiiEM/kWegSOt0zKGtW8gjnBRBHv3cvusRWYY8fNS
iAIei6ovrCPUQIa8s8u1eBNJRp9YKMuG3PShP8DIB/AOAHxhrGOQKYjZTjY6
IQgPt2Qd3iNESFYYJAZbmglZcDYIBj8O/BUXH64er847HVjaquIn/pj0wNVC
ahO35uTNvlVLoKFjH7Q4L3/Hu+mbx43LHPAS6yKQx8HQiSTGbV6OXZNYCXUC
MefUbj+HzLdlxGb6WIMQ79MzFhHr6866tf6sOPfdHRiOVA+j8nYYdlCiXXYn
sYG+aN/f+zVKckvoPfiJUW5CCEgH0nKRZGSB8S37z5DsBLASPgqfV7fNvD8P
u6YAaLUwSX2/LHFS+RCqDMuHGHw9D6ifmbOHw6nH8f5+HR6vc2L0KnuVJJ4l
I9qKEk28h2hBERkJ7YS4SL7Pt3d3f19BR8pR8s5XFcwKFRfKeZxEU4e+a5Ej
7j2Dm9VEe5IQIO2J6EDxRff5BVQGfUChv9r04d8U9fIPmPPIUFxml49c+rv7
t12bjYHOjRhI6TM5SmhB3MhHyMAHtOKB87bEFUGXnx9ndEW9qjci422AQZjl
yhlyrHdwCCAJS5dfb4+rv3dXPVYrzMxG0EcerlTj3HNkEKdTH1jJ+aRy1mBm
dbDDaNhXJNA/X7Pd6vKdtBK7HUwHSYfIv9vxpioe/zvD2PdCEQ/Hzc3Qg7vV
TvwheD+34WPSv5T9kagVA8fZUROKFE5UP/ogj9xBfSdKQPNHMBX+lCO0e7h8
vha0+R9rYQbZ4HVPkon3V104uULMbcGCKpZ/hafpHy63jizMqqqv3wkhJCHe
BX3KD7Xl9j9UD7vJz4O95jO0Pb+71buj0Gf5f1WgzbzX8hk55dRxqeQe2QcG
5VdEGBkUJG0tYPtQGiIDXiokWKyARCCL4W+KSJYcqZgtyIFSFFgoMyx9oh2G
q4iqEmj4LGam26qsQ9hTZq1S7eZrH4zZwYGCUzb1iM23AXCyiatmaXUT42Kp
Pcl1lJbeqUshxcE8yZCzBN7k5NzVgKRkwqZsjN+Vhhwm9U3ORdsmEzc7syZr
EwN5TRKSWbxNxdDUE3o6hwC2E3kkkiMCBsGDKcgi7GKAyZCx9LJvvoJwRa+k
dSZNxXFKQ0bwNzIp0cG9syGTJLs06mERd+hNTi3IJg3m+yUXSg0UjNvRkwUY
KUkU4JcLqSl1J+X3w8atUv4b7vJnpj0/MzkwddMvgRA7fkdSCFD236/afz1M
2YN35zmczb0zocjNzSWlFEX+T2fxY4+DwtzyePsGB+DPfEu4OTm6n2PJpzIp
r7P3I9HB6x/SUUewfu/osOBu3M1Uascu13ZfU8B4QcxnGXYdUo8srR+btETV
mihLiegjCh9Iftx9vmHAnMPiY7qh/EcEYle4foA+g6zejxKHiHz0xByp5mB6
Hrn4XMz1P1fhGMP1/lwIk3Zgp+j7/lj9P8PnqRGCozdmmHP6vi/UBMD2ylGI
ggswAAsg/wo+/9vfz5+3pO7u7u7d3d3d7qYVBvJs5GOEAImtd6ljVyBU1KEi
HBDrDaELaBsRFJZFPSuhWpmDAXOTJqRCIIyaF3YYSZprKqRCZ9qpwVgcQtoM
9Rppw7BmZgA2uUMu4e7BTw5FBGsaHFyMrQwq2BsCLFl4aLE5tyMjRWa7kITe
3VWSTpVy1CQBFmK17nGVWGA19p3ps3Kig9Pk6aVtBTYICLAYBclOryRYvZ2J
fMthSlbDBzFSAi6urE7E2RDCsgEvmViCt08ViKuMqgIovdVTPVyGG3mVtRGF
gMDQEKuNpTL0KBETe1sLGqnMuHYATizVD7jA6XRl6i8e8wAG0GD5QvCJrBum
BiB3YtLHIlNADnvEf/gcv8n8221agJeIwkZsCS58VPv4845pBXmgPpKqtKqq
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqvq/m+uwRJeNAVqgKS2lpaWlpaWlpTzY
0tLSkRa2QsaWlPeYkY0tYgINBZZCqKSg+qfB0zcpheG0XrvboYxSpV3L5Rd3
d3jFSJKISUkGKyxwd50Ctz30UoTx48VwZ5OJCPgas2AwrqllpRYUTkTdeqzp
VLJzby4xU25WiRqlpQFLZcjyVQll0Qwh5GUdQ33oo0E8DpKzob8g71O71sim
CGZlXZDcGt8Hv356Vz1t1rVZuGgrtlrKASCdWlZGRkZGRkZGRkZEQgO46jy2
mIpFql7SeLI1ZWCblIdjv3uvIanXfV8dfGsnUUXY7ZdGt3h424Dv0dRh3AM7
4qqqqq7ySRV3kVVVVVVVeJJJJJJJJJJJJJJJvJJN+t3JF4DvVIZBS8VVVu6b
vpIu5wUdCwCsAK2K51F2Q3EA5o43i9Cbc9+klwL3sXM5MoC9iWTpN1447L2W
6d2JTTKYQ02B6a96Ds7wYC3u3Vyd1VVVUu5VW3dgSVwqbSpJRCVpe/Ivhd9X
5byuiaMmk2NtVdG/Rb984b7cxH6mBWD+ZxDx9aYByFCnp8vyA51pf1HHHVEk
ML8o3eIyQkCX7OezMx/T15TbubzDF+MgAlASg39tuPkdzYUUvLxjxnj4wg1f
LG6zYwBvEcwzks1SfPrYsw0Q27jtRZrBuiySiV1GUI/UkoTGUJGUJzJqJDPP
u83raytvNLeDYV+/MjvxJz1sZz4zZB7Ga/HLheOoLiCmEVoTOl92jKzu22dZ
tzqjRwyhIyhI9wSMBIwEveWCYwEjASMBMYCCCaYCRgJbATfeGAjObsEdYih8
Qq7HszeARmetqa5flL117uOeIz0hnHI4sd35zvg3g4ucPmQLsWqs3UA0KU0a
qAaFKaNe4F2LVWb7jrrBkXhzqByxaqzeKKRF35065AyROi/fWUvXJ6Dvj+Lz
lET0H9VJ5B9o+vPO/T179eHr0tPfruV7Hvw73FAzM5U0DXM6q78XMh/Ucnlb
PvuPTDnXu/dB+u+/fvUkkvVzlLrYf3HDFYvfufbDb9+s92H875Xv3zke5tV4
8K+QemBDjpMEH1eFhAmEGQRdgQ+e1s6s2H2OuTse/UeMN69bnuw/nc+/erPX
g8fkHa2Y6AIIXRwKeqHtiDHk0WkGOungWNv1PUQFtdXHXiTnza8uN84tttfe
/I63hmlQ6z36Qwggwf/NpSGa9s2hLCmEsGptJyv7yIjBLdIcSX2shK1enozg
zH+VNxZogiWWvTxOgGy0nzJClO0IDpqqDp2OdB0oybG+VPyC5Fvwwj0jPqzb
6YGU9VO/nd93VkadP0YXkYde7Rhn4PSTw7045LXjixB8px2wIUvSIXfc2ZYU
0qgYhJ+OU52nxzDm0zxEuXBdJwTX2kthowQpecUywzCcF07gh/BTARBNiecx
iplv4cNBZiDKb+TDmvmZfXZH66ccAuFz/cxypFn/H5zVTwcPlAvFqrVt4ETU
5x058q6T6KQexD8YxXACKA3bx0hea+5IBcK7J/GwH7P1UUBCiHoKQCo4GZmY
ANM1EzNUuCaqhUs1mVjMl3ZG5ZVy5JN959wN/QoK+mANAq6weckGEFXQCAKW
JOsoRHvSMSxueOxQqC2AhQVCQXod+1XkWJcX5QxqgQKzF2a4ImFRwRMlWsGa
RF3KkhdCl8aDUHgJXuxUTyR0uBdwxbooEmnlBAgXJRoRnfi7vKeeENzcL8uF
INRh82g2iejskS0xTKqtZ27bcr9kGlINqkEoaBq0fuDgAH0SgqqJaPSZPChR
UpXWIYNFG6OYpBCROO+Fpm2XFArMMaujfYjhi8r0oQK6iOo9Za+jjQ3rNMCi
JjZbkoDQziYkN7gEog4H2wK3m5zFykHzZ2ewGGghgfv7INUiUjRWwBwRw9gH
TqX211b1LbvOLLoInNjKmTWmq1QZa7HC5qEwCVrffRY3yQ5uc3XKa4lERart
UhPL075pSJm2rOPMA73vxzcm7AuqDIBIcKoRTqg0GM2rMu78OoKiFy+B0o6H
O7qvnDCEGmGiP+CgBgR1Scy486fuWdyIKYjuGYMHkvl0HPMihQMhph2FzGaU
TN8WCxarNjuIXBAkUVbsrsgWF2FW6r2wXcWUqPuysvzyFkPGj28uuEIMlda7
2bDA03X0RVC0MPrYhllpFRaCYYYxxgQgQX+EJLfS8QCMAgZfyJhF8nB9JbhV
hnZbEDxg3iRzpceCrjdyTB9aCdmwWNg77yhxgBnr54fsQzgswBLeHlVNd3ku
jH43HjHF2fj5octmdVYQ7EB/VPJGnyNiJYHwln1cGYUc1Mt0hqgw5DDMTUTY
4ADkjFm8VBL4cquh6HAFVZQJ2OEeM5zB0dmHbpd0QdJfawMgbh9pxLb1cNk3
kJqwvBGGnyDFvhh6rOKtERmoPMoP0OADbgi3rY6KYEOQ4aKFv1VNd59VBMlo
twKnUKFJQVquATE85MSez2HXVY8PFiMPXTIFgAmm3clpvodCACaM1FCHIvYQ
OOHqlwi7q7ka+csUavp3ULZN43hj1MUx3784l8ywIyXGnCA8NKthpnWRVg2a
Tcwn6+nungDnfhQ4Yt1d/SpDTXxHEZ6MIuKIURvjYNOwITfcNxyTHGbvn6VR
QBQcZx6LrTikbS6zvIqMVKq6FMwttwOxKI5jmHQLw7LdipFnQIc9wtx6Om8d
DHFW8AwRxhKdnjeFMI2lhWDRfCkaxcq8NxkeixAkB8NBzyGRZ1I6Xsir5clm
ESNcZI/qrJhxfruxI0JImUixHcbS02lu/1d/bIjxkRxNmoGwMJc8xPTlzL/9
ysNJwt0QrWLxLQhBnxd4ELTZVoWS5NOFl4tijauo1btTdZAl8uYRFbEbUKUZ
u7edGzl5EbOPmVEoin2ZKurh9wvItbx2kak0GywXU+ALIAWPUUpvjnHY7bWm
2RaD9Ppt0tEck2cO0PQrBoBA5CwfDDpgPTQ3BAKq2IKO7KTwfYrdYEy09nQY
IsjOjvE2E6yCwNSAskEaBdjEoe+9iUi2OhGxAGLqpQ3OMppfG/mVxNATJaVm
3btaMtczka5Y0J4qWFktWzoRSnMIoiSyzsBksRUUBV0SSMH3C9cl5h8qvOvG
dpLN3ATtYkZrBoYJgHBWjqdrCWgGiiBlVs8FLCqFcyhgFjIBA5oPpVBsZ6cy
o9cmaIsZuWyY11xyi0rx0nFrRbDS16PSgLbFGUGjBEoMQkQbYgBGeBrbnzP3
gkoHclQJ7xBsEQpgk+ewIWXF5JYLoRKFSSSKUKKkpFFFShILABgkYLBisBgj
BCESAilKXqy9JaShSkKihQoihYLAGDAYbQRoiF70NiEGxABKIjCEIQSKsCAV
KAgwZGEIMEYwD57FhSwwgJCCsCArKlC2LiXSSUShSJSgSgpIpAAsRGhgkC5A
pioWoShL1AsonVYsGMXkSKpeRIKKU3WBVEiyFUKIlAXpISxKhIIQAgsIF7Uh
YS1DQsCrUqlgkFjAS1pChSylohRAgxEIS0P6AQQsCVAaNKjkoVVVPvfAHjHL
zhD703gx5sWaB2v2c+3ul1fN6VrWta1rSlKUpSnvDp/mAdg8WTM52+D98uHs
82EbXZxNdm/z6O/zbEQm3H4ntI7ISYptttE/V39fD1vDcTbv2NAg34rpY/4n
45+/q3ufjiRGL4CNr9TM3qvHn5HkeRI8jyPIhY4/8aCSSAUTWcsPSFr0wx9+
3ycMyIl3hoHbhDRcXmEynxjLxSPDQkU5ebu6MiCiUxoHzVwggZBRMBndSd3c
3va/jpdf8Xab2DsFE3uaaPkKM0wm/gqmZ5tXBxFPgm1JrRVCt1i1IXqVXD5v
nenx3K0D+PfhozFHc35hvrFYAswge0Q6BkHTRflQstamPoh/athLuqKIzdjh
WeLazzaX50ESuAdEa0QCE22cY8+qoIbB1gWdfrBig2iiBiEigpGyO16nbL2Q
O217FRPGjs52I3kKZOXZbKsb+FhdRFbacM8iJdVn0cZ7bPNCZqO2hyWGMpF6
kXis4TvIFMHrMhXKAStVqylAzeDLqrtmdTdVuYkkgFAmhVDmDjhgGsQ7J06T
tyrds0IAg0ZzWmt+lmoSvgOM5SCpxdoBmYJoU/eJZhLXRQZlwtiGonmmArP6
j7g+4FIHR4eeIbVDq6lQ836IwuQQGa8VgJwKuJtRi4qoDTWMgQPE82A4Qg48
85AiRqfcAELhEmHpyENy6AgguYOQ4tdhQJPNgkbL4skQWAYC3/AAdT1kj7+w
XnlnpTlcaLpDwMAGmc7gXJMlqTvArKgkCYszOkkiZBSxd2tHmI/LtalLrN4J
k0ZLc+cdN6PMdyjWPQLG0gMGeSPdcMVuuHVvTvd7WRPs9jzssJTSWbG12Z5I
yoC8gM4nzYoCJJHhO70zQ24JiBJ8qJeBgAYiIL+duS/fM7Afi3oANg67FXuH
im8B3DEbYvboG1BdDVWHUOOyder2u0x4Glx4CmAIymD0aU3fT/n6GNpbJAAd
szkWwkB+3JE5R8jSh3zs0quxVcs3qyYmxhRqvBPR71wSaGqNBvfPTMro9kXb
eBl6As3XIefITmuKY4CBzKcAByANiCBy3arjQw6fzDdKxUPXj7wN+71usx5r
DpgQw6ciS5Zoh/COdWg2cFoFCjaQJcqgC9u13vOxV6ecCBzgdkCfH4t4/czj
/dGZVjzeKKep21zp7e5eoQRTt32lKWiPC3A5iNvR1ExoO+ktNHSIHogMm/Hr
2qfcmW2owCBps+zvzYZbNr2949zJDkHjHgZRb8ABIOK46NRUg/B67ipeqrvh
yleJc1U91GUXt56btmvXb5YhxAZugW7HU9w2sRM9QieyEUCiy19NGU4GEKSX
DnsxfiFE246rLIifOipEZHWOa7r5H5ux+fn5v1P2EfkvUZ6dLyGj4EjYZI3r
IEgQdx7Wv7K5a+3LQZ4PE6BinidMI5QyaWjoZ4gaVQnyaVgTqsBldZY03ZrB
QVxSes5pBdVd4HtVtWaMl9rTRWXNTl5GXGVgM7FzE45GqFombBjcBuDmaJh5
ozORtRu1tPP7Ru7yE73zbuT0KgOL83tc47v3O0K4J5OG4U5eQHh5mqnp0NwV
t1ez19OhcDLegYayIsXxYnCkSlEVjuXdtRxtw7eU3zMjUolyhjUTnRl1ac76
RrWaluHCXVd5fXXyV8O3adns/RCKnc+ZFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFO23AZgooopCQUU6e3XpvRz5mH0L6fW+X155pNq4Z5G6nBkRYS
iFu36fTb6R9N+nEjrgYKyMV87I620o3jegXY18tJeX2aqkTlAQwf2bpC85QB
HeuJTUJFwhL3GXKyezeadT6+4t45Cet2WS7tJVsrardOEPWd4LucOWz5HnGS
eiiwWltrVa8LVaqoqqtVnUnv4ewtWa/TuUgAg7IdQmyoUmE3yeG5l6hpG191
YqukMfzSHhLgNF5l4V3tM/TiD2zS71W52GAoSZDvRF8cmDpWv8EVD0ZwgBwJ
I+xfrwp1GVQbtuB9Q1kzW+XdtFOLAbV+WRPKAwgNA65SHT+Zk9gs2AnrgEd3
ZQnxsrvl/NyO5wZE5y+2/p4n2cd+p3tx1VLo06z18N7AnnrtoDN2vR6mQzLC
1Z04525I7I0sueZMgQ7twk9o6GWNmVH4guoXgI4jPaNGcO3PbjLPDI4nn+dw
B47lBrhhXdCpg1pdVUA515lzQHcIWwZgRIMF7cH+Pq3My5fHL2gLpwJermRU
+ohcTv6swaswRx5G9j98AfT57DtBMPmiQi/DOOio13wqYsGUyTUYgBXOtqTI
kjrkBfcOh1uU+qwR8dB14B5fcSNDFbBiJqCUSp2rnS+PX30zOLsd+x7JxxRQ
tJg67hMhDGc8aOQbKdGIxHKSc/XOnvYVSrIB51gkr1EhipAVQTWobZWSTFJI
OFDpEIxRLcd81iYURxFEeGNcTF610zNud9eG2NtddMDc4KiBrqUA5iqBiKCy
2NLiBusFXGaRDSCKGsHTTLjbe14cwjDpg4RwwR1OXv4SU6e41UqVAvV3wACP
OB78Q8IOOwRbtNDnpQVky6n5n8PbCgwTgMgBBigtS7JmMo0UCLCvih1oMFK3
rG7VHlCd5jR4eMlaa8kMTZOgAIlO4cpYNoocNaUW0ZJCMJrMWWhkogcd7tm2
/R688qvO8jt04ImuQ7SUFGIAB3EiDdo0c0WIu3UhnxKmOvsMUr3wxzdpGiZm
GTSdPAbb8sJzjv18b32mYl98WHsIcNKavRUNiTa26llvF40aJY3haASJxiFS
oFQYxCoNUoi9E566r0wrSG71SuOeIBzfRciEhFm5t4teVE/f7+gqrFXAlu5Y
d9+oiH8v3HqLF99T1ID9XrTcT8Xu714UTcT2JiBrxNvEGFITvmklcYINYZ/m
+ae4Z+TSRokaXR6erSf5ZXM/h/j6dbSmLVR9L594qcO5XMqecN0quTV5I0XC
3DOZD0cRydlxbk5URGAoXiW3S18bUcWW8Crd53N0KjVXDh6eYmIk4L15eaeX
eJDxtXWaXyhJiFpJvTLhgAyfIqLeDVmQTFjHw7eKDs8/JLCft0QPqe5jl8/W
NU+Wsp5OY+/PGqUeBsLAjdg0XKHPeYLtupy4k+1yIWwIIfJ83s+6+YXW5JG1
b6ve8OuuXgBboEsS4b+cuB549/h5PhOrp3n1KUr4QJEkXbrhvxc+bDoq3zW3
7raI2/6TzJgUP2GdZa+UZ7NMx0BIPl83AwnPlSHe+PbMpoChWJYvVEhVWha1
Uz62eDO3qJF88Uu04BtFqklmCyEBsKoij9tiEQUFBSJEFBRr6aFpYpEjQooo
KCkpiRiLW5atS6uHRtakgSQRQWhaFBRJIKl6uDEToD/EDIAgHYyXI7vii/BW
2eiBmIxxzeWcykWbaoxKdJyrJ5U6H8JtR5oK02SEb3qNyq+MLYysjWVDjC1i
zGC3FNa1vmIY4PdPn7cnbgN8dDB2/YDeSB8bHo0Qcq7vkXuUoFJe4FToQlKV
rRs5et71rKzRNSS6tLZ/WvyQq6+IiWmqTjDnyfj5Pyh8XNdKdzuJVZEBh+81
dZHM4t5FKZ4SFqRHnI3PSWJKrRquBgVEbOpkNOK6FpnnnbKSB6HN9X3e8nfg
Q06/FkXNEZBi7v41hm1ecx+ce7weM2hYxgUIJCJxk9NABCCBkqd4ZzvLOvWV
RnvyHBm6eNFSKiT6FwBtqcGJbep39bOks7zm2EqxIFYy0R1SQxmIzZJLdTJr
53VDG791eOMh4d7eGgCT9Sw2PWXxnNKIYIqc9ic4J1y9Ta2n1mF02P4GLDvM
zBQd8ciiJt0JLg64QPUUppJ7vT8QYdVS3H15uBBvXe+z3AeM3c5cXcTpM/Tw
OMbeZI3e+8c2RbuKiN0x5s0I8Q0LrY2JfyXL3FVD9koiKquY20HyJ531d3wt
fa9gZXM3d8mp1+mgY7jHI9/n1huaObEM3vUXhK+sztwaqZyjCyX9jxwc61VB
ud2Z/I1BOBquJUiNiJu363Fd5CsVeY4Q0WiqdB3DoW5Ib1JfSt1sNMstNWfW
kpYe4jWwz3tOzlm+cmIuLUbM0kHsSCoh6x9x6qXRKm3nMJUSodPu3TzsVc7g
eYUZO2NaoNJ5WFSctQnNzaq8m5qytl7k7mjNybRQ0PZd9sF6y4epNa9J1s4T
Fitubyb24gKax6jFOd2bJhnpu1zBjXdplxI7IdZUpSUlbday2IOo0UM8jYTn
ZIRjRy4XByRA7TAcHfQgFIGcWa5OGk08dHk5XNhsEhFq4+fft82OiCHPk/Xj
tPWsdN0jgt3uf3UnCULPThPp4x04uswgvc1cZNMU5Gmu2kpm/O8Na3qAIQde
k2aJkO6r9sAlHw1tfXebnWzMZJd6DILchIrBkwLljcIPQ4gHmVUsuGZT4Usg
jXtQT0HYBCBpl4pB1DsEQTv6TsEWPKe6WXcp5MIyk/pLwQFCOmIPuL/OcFDw
BzWbBbRuVLynHxzq024NtxLtBolrR0wpFsa3wk3Vt9LVxfCeOOMScsXD1cHz
xxJz74Z/wbqAMAMg0AW4g+8p9qQCPKrt9W8uKEEzAvkxDNAp58zOBXWafuAZ
gGAL3iI76PpiZzuH0wOy7xkM3nXV98ZnArvJrSz5c1Bz5BfCwe9D00NLegC3
jDrh3rj8fvEKubj0BoDMB6aQC2+DYFZwKeruKlpIcs6jjoSa7FTGavLwXeoZ
Du71Xmoab1btygfqZHgZix3iD1fDSnlDwdBhzvq3yLwcd8pZCeepmtsQYUCN
vDO1Ro3NlMGofVdQOtPXx4Jq63yPKA848+bKePkAAvFEY/HeYOjU9zPKGQZU
IWKZ/0NHu0AA0Au4nlvEHeztUFLANL0nXHg1T40qzPtXYrJh++XFZr1eYsPP
pD9c5zOHXBmLZpvqYAl52GZz1HAryuBQ20J6Zgw6fH2Q56NzUmFzrr4t5DfM
UuZn64YVFP30B9AF55b1sTy1bOJr0kJTiTnJ0iMZYQtBsoUbigN6rGO7Ycve
95sRg2FZTg9VxlgIWwYlK4FwFTEihXq5m38xCxFeCxW5AI2VvoMGQi3HLDi+
cKwaN5TkUjg2INuLj9RkkOFhEXZmGphhd+6/MMmDd+Z7hPyKKicNyL4KWPFp
apqpT4YvMAfcLqlrk5heS+QtsbSVrJgh52HzNqXfDUbEvoKR0YY3Yc64oXCV
3D26e7jdDiH2bh1WCYuLHM05e5CyuXxPTmRzKRg6cXtjXYm1a4kzmBT2F06C
HOAA05LzB13TimUcGpgsJnJgsEsm0MmN6EuhaOPBlMG1SGbc0ZYCz8OMept2
L76NnHLFfZ4aH3mhbU6NbV8343j6PMNHu+Kvfg52PJLtW5mTIN3GS7ba9t4l
ST3mYUTLkkkk7mUVBoRKDzr4gmkBaBOKKM3EO4SDdFHyHxbPUckGkgboQg/b
OQdN5OdJLloYsEmQGFXQGBYtDMcu/v0NmAfr8Zs7Sz17uJ9YPAF2dpwYBr0G
wNfbMzLvoJBR4R9HV6mDCBN152/QSy4ADaou3iLYXu+YM5l+BUN5WIrDI6NV
zJCvsRAuQSLAGrCUWtaoJYFVKJdzFh5PSJBpYPkAR5VSBWZ4k7/AIfM4PBu9
mZzvIrq4HOZfdvGmuozzTIgMFZgkGggK3hcznJ4ycp8Cc5huUBa9BMEwFI5f
kgBWHiX5Xpc3VrXJnIvIyH17N9gNQZyykhewOwH5Xdi+ugDtHU7kDbU30N+b
tR+zsj1so267MkjsOdep89AYIAanQG6QM89DtncQz8naA5kYqt6Xj1kVtxJ2
e0XnIxxkQylZlvmEADBwzQuCqZyz0jNCfO6E+g4F0OBC+49QLtwH9ZyAJx2m
HliWwszggTzlZ7AArkdYF0ty9mTdQJhXWMyYwVcEJnN6lMrHyKbKgyCWtlRV
ONBvEagceNIt0gScYJaDiEiEi6aUgVEKhI504aXA55oR00whYkVdIBrKNhBQ
DbNrK7IYlRBaM+Xy8umHY3dc8SjvSEbcjjyanMALIVUMYNowKCBi7XYCTCYS
IGNJQGYUTlEOWnDruqaQVNIYjz01ycMAZgSZm8Ma0riASOJjGMXC0EJAOZog
MiA1D2zBgAwaZubh6yYSxTR0TMz1KViHVEdohzi52hwvcXJHNsWR035WA1xS
1pQO1qW0WQxNYWgjxxxsF4rkgC66ZAMBZYASZUyHdGuc3q6dbU3Yiehak1nT
k2JEg2pxEGqR/JW0NTVnbKzR1lpDWtHhHKub968Wk75evN7tOrxZdnEDWSsp
8WicequwjeY52ZBjIrdWZtaipF29zgjLBy9OmqIgw0DZRyVUO+rYrIKh9hZQ
JkWpOyKoTjoJThUTQOa5nb1TObCBd0JwXkR2MuIBM4bhtxAvEbGxFVcvByDg
OyiZF1SJlBOTZJlQojEkbrNiz4s8UlYemODY613LZEKRmZ6p6pmY7dL6cOy2
9rmq3SIyAEYbh6djQQONwMc3NLEgEOWhBBAOx8+hHz41dGOoVfs/L3NdjvXA
eR7g7HurbZUX4KGODESeqE9EyHk+zXJsVZ6G53D3Hi5R7swLFXdVVVVI06kR
NMy9PGj1BTnpweYB1PQqjlThypEqBJ1RLRaipe85bZsYMxzVDISMgSJiBjiF
At4o2vQ2ikiFumLKlum21jp5NZptFNIhvF4R3tQvSHCKa7WmMT9d6ow3yoiq
NxTBvbVegBBriEBgwUR7At0NkeUQ3hm7pbQ9ubKu1Xs8YL+eJpA242yVcGRC
RbwR9M58KW8dIiZk0l73AcwDheg5pAHOua0uhNSkDSCnOLhpErVxlWTaj3lF
Ro9ZjmCRVhbkasMygCxZADoBye5GiqDUQwwsGghHcEdQ4K6p4wSBaAd2qj6C
CIdoRLnGTrxKK50lQrMEc1QLaImYqYuUAsimJmLacyBu8q6D96hMmHysc5W0
bBy58Arr6F3se2DdeIeh10H6GvUyPU5U3dKK8i/UDMyhKVvajVbRd5yoyjI7
haFILtPQ0EpWAaWBltmQaiUjQ6jfL14HrgdxtzT2N+K5tK8hJwyQ6A4H5Rqk
rDavN6Enmhe8LJLAzMZUrawCbGeDwyRey06YL5DDpoF2+ubQA0HI/EM0y44C
y5frnC6mcneGqXYSHAwcAgEswPv1pMc/adt6cSCOiCRyOIep3LJHfquF6nS1
vJVSKPD9gZqXf26Ks7MPS4o247HpmYDubuxD7nnfBYu8rs/AjOa6k3yFlp/H
7S3F1pq8vu9HNGc6u4jVQ8yD3vlnrFDsji1AoAMOkncYBwCdtIeuzToP7MiY
8IsFCfU8bSJmpQd/ySzXfxVG+tnk7Qh26t+ZUO8bk6/Lh54+0oWM3wRzBz5A
GCu6RAPOtgkw0mYjt+zQqahzDxHdpGAIfFmQ9VE0awrMQs2XzAgXMC6ld53W
ihN2pBRHthsnZiAiBmlds0jtCokwDHgHwyhk2de/hQj7n4q7E5bxc1FFPhj3
aoTlxJ0G4va1ZuzdpY6lZJe9xS8SjmXCut23oZmZVrKqBaoRt4+uVu3MlVJe
xtW8BxeSdy1hMJZqpUsuMhB4T1lVeZW4+ZZm06B1bquO7rk69ZGRThFagcL8
eA9qbkTQaIM0KoTs2duosCo1sUXHamChnWB01033ovptTJkdEWcRvFO2e/Wt
2EzNn3N47qzxhTCVJXZIQAgZeMa1NJL6ERcll5bGAsbe4bl1sr4RQyYYXWRU
fNSKUc3aE0+33gr3wnI+5p9Q9pXkwRdSqVQ8DswEtGKjWkqsftrhBghqHwfC
Phjv6b46TD5+fWPHPy2rc6gyziAW1tCa47XGXAKg/UN8SOES81ssn4+y4MLP
IE5spNMsAKcWgm1FvDUdVpK0VlgsDOq1SJljSXneJvJE1yh9RcVSYGUrLfGS
EKeVbA724qKyAOytOocy5dRBkiMpAXNauY/xkCX3ZfNIpopIaRbgdAM/R6Ij
lxEnBrtVuAXtwhRHVlxmVcWSbiuosECQ5fos1bvKkaadbdKMi9M4KNXgUib9
C87YdsuQBr76ArunLR7FqUGHpDJXluhLIPyoRwg/AdKjyAQCQxjZlvQEkAwa
Oe3EHgO50mcqeLDR7V7l6OncreGuXO7djXAPoAegNAwV1ADww2cXhwZNQ3Zj
OpZoIIre4ksG0GixAqE0RBHT2nl3esmJjpwunFgkHgFTbw8uF1WiMszNtjGs
IErTQtVS9shZBdL3AG4AzgxnZ9GWjZ80WiJDSWzcSxOWtIWccul9b4CZvYMw
oiLSChTAEbOOInJjbnc3EcmTtEIs3oMLK8AWdulgWF7Q1xxm93NG0gPRqImq
gLRSjGlgEHZqpW7Jm2dV44dtw8JW3MKklKUTaWXA7S7kinzEOPEN70bPPTW/
066aAHKIyIOrEq4dEhg4Q4Z0vTcJpz3wxevezNTftuWG6TmBOWa3qJTZkCbu
2D3NhiOgal4AMY53Y6Rh5HMmttW6AfiH64p/t93sMMdudokee89D3WhoRCN5
TQCYCh6r1E73eTrhBe8fXeJodc2XQPIiUJd6mjF1BxdN7xb0OlfVLrqxa7mz
o6nF3xkQa5scQfJYwEMxYYcjMlULcdTozenmA54M45utmcwLOfA8hF5ZpDWQ
F2EKCC142akCmbvGMmIZZKIyzeXywcl5WOXnRE2o2HMh6d5W6opxTuMqZutg
zWrRdUjOB7y8SNvU5lQ+QqrZO2NJsW+Q82akarTwYFGKMF1lJ3k2UCpmFM3O
l4uhdCqNVlvO5AsI6ImaVIZeoRTBmGp4HKcLGxJrfOb1ttrZHYgLJ1DT+vJ0
kktWJCDFy4t+IQJCAJRzeJSEYXI743nrDXOLNLpaMssadeS4fIGWvEpUenBk
d3DdaJyCtNCMqEJ7uA/CXE8X8ZPtO4Y4zM9uOnM1HrCJB2lJniDMMsKEwaQM
IqhLRAQccVX3CHwR8AoJh5Aw6GvdmvufnLCYwI8dA4ZiIwaBLS0cGgkkn2Jl
HKDisA5WZ3ZzKqq+aSeqmqr1ezPIrdcIHXjqRmOLqaEEyXNKqdFJQkadzUTR
KyKguDPcoV+AIYb9/6pHPqBz31Kb4VEMKdqpepdFaKzM2pmycFxOpOAdzmqd
XaAClRFqonKKWihaERSiolUcIYqqqqCEhEVEuRjQY1ZJCR5RM4ypmZsbTQi1
Hsxv+Wd71oUdK4c75zprVQzzNzZHSzlJcICYc2tnFUm9QSGZIEzLF3jnJrZS
AmyLs5JwKRvSITFxHqwQs3YzHIAb5TsDze1BBMd2u7ckcPDwsB87NwCtPa7m
ZSjCagieD7AMrXBAjjOAAG0szthZqFiMQ6b+XZHfrl9irBonf4pziojrBUkD
OVpOMW709mUnnANspZ7qkyOo8ZvYUnZqvqRDImYoQxI5D0bN399KZCSTuTkD
qYOH2bH5Gnq9+3alx0ceh5x9zTA+eNW2w39wBgiCQW58jkw/G+4AIOAfJ2OT
0VMedR7X7wKu+cho13LEsY5MNMjuDdTSrq/MfN5F/dBNgADeECIHF8DF755c
fpd51Xrd6295eP3wORhHSTqVH1gmR77QRiasPEeg64GsswUOA8kRVoOSbXrm
CRvyI2b0r33lxfUzhtzRF1yX+MCSAAH2rqedgJhfR/DpRaE+3EFph4JOGJeC
TJhOO+RmCWUOHxzxwETtoYY9Tb4lS0vS9VShw9Wq9AfAIPnTIMQNp2ft2lgA
Quneccy6qdkTcOn7DBSpxRBbMu+xaqULd3ww/cK5ImnRuWG8r7MBBILEE16A
dm4ND9o2IPTPIHou4L8L2XiE4XSaTkZ27DpiDJHnSuWtOoQU6tROmul1Wzp2
aiWm2UQMGRYa4vTrp9Gy78svZKDhs0ohyTZgIy6YliQdkJFox2JgtBYVbhXK
Rh32FQAuXqoloFKxCBsgy8lh11OoXnI0mpx+q6qV1B0VE4GfA4BIY+hXnLBI
/gC+cTM3dVx0wXZjLzzWZhSNb0PF1yb6GelViQpgg2IYKU6KEvJokh4o/AC3
m8zh8NTUKwdxWIf0wFMACOuVdG2U4XSk7EXKqknTsiBwxlIdm5N7MzGHVUJ5
OjdN29h5GTCqjdWGwfb7Cu2YQGkkE4yrwRCiQpEkRNDgNnDPlS/S7s1kHZu6
zbMzWk1Q16iUMm32aV7dblDLpVafYjYgkOJOE1GVGPcxNTmVSE7CSnN1ErVJ
e5ioy9InBr0aqsELKMp6Nuoh70rZ3aq7mYmVLxCfNvHWvheHBswZzbM5dRtP
CLi3wVV7GbcZeCMFxIifkD380T9ug1pe6UrESKtSKRtkVcZr1hAOpoRsk+j6
d+t4d7yrGFYpuGzHzdunBs2u1xCuuPf9nTiSNM104tbF4a01qZNJuUV3m1UZ
dyMkZJCgkHkv4AN/fHgThI9CCQwG42MdxC4Z1iJiMJvzvuAPvXg0murcLfT1
HgfCMmUBVuPLckS1XXsxJl/e9vJquu5slP2fQayyqEwA9tIXf48vk+Ww0NS9
WffkWAaywpPoquJoI0wR6y1gAY5W0YxPPJydpWlmiuTOW8cex3XwBHYA1xpO
ONPRCI7iFsSYjHDjonKU4WAc1b0qEk/XmZ4XHOubQ4uTCXsQCIr2Kpms+EGf
xboZN3HUdkDTU7OfSat3MPtuPDpRkQRtXRpPyTWeiJf2mUYOikkNBfg7SMEV
1eVOCXsgkC+3aby37PcXTK9W1Lz5bfAHsMGHpCc8frrvrvofhe4Xchznp0C2
7X+AQPgwWc91aBIN559b6q1dR66Tq+Dh1AUAtUWYRoopw5vL3BIFliaix1Ai
5Q0idxNEvJJnM0vT3WQ+5NEF4e2gb+gN7amRwjmHikvOvFTH0Z+yViFHF2gJ
2ZiYfU4EERVBdg4YufDSuBtBVtY8iEyNSLDpjhBonFcWGGWxrxO69NXH22jE
zbTOGsNFj22Gp0/Mu+LPgKbJviHUOrhD55GVIuXdP2QZdfDi7sXwZ3pSlXQ5
T0qm5QIl5JDBvYbjdy43qE1q02yOSufNNlOL12WyEs2Jk1Sgg8m5i6bCZNQs
zt8l6zqHg5AhcND2Ew+WYX85xco9P30o6erqC/d1dZs3eibqdOmKmyiKycRG
xWYbe5OLURV2RE3hDrNu9zdqFSqt1RqzYUI3AWSKetl8u8gbUS2CLlyIkJ80
RKtQHfXzFMXmYIuMo5W6NmplK7mbyVG1VvuQEajHx7VTEUI27OewKZmbzoYc
Zgp8pFwaimYPFjdcWAZ44ZHOWkqW3UGYeaCm8iS/bv8XyVA+sjBnqgo6koHZ
CE/He6O46/6P8/6T7TRAS5186KlSzd0DFmWaY3GgtBjrcAMG7P0+/72trVYO
/qJJJJJJh4aQUQSSSR+H6xR3D5/+QDc/F+KSSSSSSEVVX3G1FHtTh+rkDpc/
SUbLO3Ng2Jr8X8qp3mbxpHBjIDH9X8vhDHGvVxRAvngtKf5bqPzLHYIvJmGp
IE5CFf8I3I4bsp+GXfKZLE99zr0/vGOuIj3fu/ugbmSK0TsfBxH7/2f3nfMK
dxyUDCLoOJd8gYPgIlIh+ljb/mG0LUk/2REgLxOgwB6MzCysSljS170tfLlx
5cqSVKUpbbEYLBxNOHVSj1NLKwDUSQAIIiEjdqPanx/8KPj70Iucle2dBGAl
4xjGQCG4FkE0W9Z7DdUx8PJ+oLo8goulmTuPnrcqU6Rg/uGiurOFarXgORFR
djFwX5X8P7Pz/2ZrIwOC0IO3hx0Qd8u2fUMaZ/Z2gMv0hF9PSEpG5pJzjRME
EBcVb2eXlp5fRJtrrrrrrrpDSGkNIaQ0hpDSGsNYaia16RbB/xMCqgOY7vN7
ilEENkvhCiQheoItSSKRO2WJBE/msfouqw/P5G7XqRIBeXNdckzgqEgnQV13
sqioHg6DLDRQkEOAXDNuNlwBk1jYdlXIAOEVh/qFvgsxKG6aRE4gnf/Pj9oY
D+lIQrCSdLM8jzoX6A6RgF/OhLpDt58ucMx7sNnjMbVakiO6bK9c6TgPYdfw
bj7iPtR4/5tUzXYEo/lVkf2WgZiopGy0Zv8iHAogBm9zb+BA/hJBFq9Bg/uM
gEPhdagP8eRtrqH+9ykgR6cQ6b4dyQf6hPgA2R4fx+dPMJcFuXxX+miEKRPR
0oT6+PEC4n+unW27t3CJmzy5cDozPYo2pHW4k/V/IOV9a/6zEbU3NK/m7NJc
PM4P4kBJFS0Dz9u3M5lGqqA0Txp19tpPNGXUTh0ndo0hI8lSZkqRWT75+7t8
s2XVxd/91TD+YaJQWLadaj8fwQWLlcqSMrHZ/y7O5m/3yIEKB6il4c0b+0wx
Xal1PB5suSXormowL2DAQKDC97L5G2TzmHtVD9EPe4zzLPzRur2cUuRTpaHT
6gMWwfUdpZSHQoT1ve0ZgdkehFpaKFui0qWaINUC/ZIYUKpchGSgwEasJGmJ
IJIMIAySDjyUA7GfTDznzWNJFZ2lpaVDzJXaw2vB4npuaMMnzLCdqc/terwy
VOgfTvx1KO7P1Ll6R4Ue4aWPUFxKiW8Roc7wmGYMeB22M9wA/PDbwvzgPucS
T4oHXfAcPd5Fp94jrD7SgGZ1CiQHFvuq5+xSJ4T2WR1qk+T2l9TvnNHYaKRU
R3fOXbysyKTakbaQ0+RkZJPX61jEc3H6C5/FodnS2Z+k5rSPCA60jjGLl/+3
sTpJdgeyUh3kLS4RPLqSLCALESl3JiHQI6rJOwyWn12tuQ0WTx81vOpEZ5o9
BOSpoioy+36WV44j3Dufb7cNp+x7ct2OZzIv3d+QCldpgyA/jSl0gKDUoMuz
PKPSWDyDAOzL2/LYalFGJ+6s1wOwGDYW78Ui/G6NjC7z5PtFdx+TyfoDEOqf
63VUqlrd26xJ9MfKelcSvf835FY4ryEw5l8EjZIkDfAR4TFgaynFLI9YKXYR
MtXBeJP0qfMSh9zu07kst2Z2Wv8rvvfHezMJdYKkdkz3lYg2DwLlg9xF4RET
rDukqhxIugz0Wk/K4dtzSUnOmbSz71B3ChmpEcHjTv6vJ9WaaJXSu11aPixC
BD9b/283e8/VlI+M/gsspfcyXXoy/U/0s5ELZH7Xd3BIQl9Qcw+wj9rNn4c9
Q/N+v9Nvz/tg8MJxmzkVRiH9mYf+qSSST/XtSn2sA2xpCfBJwKKObNw2Kpma
SyN2DWbd3rmHKglx+sBV0JzCVfA7xy1nJmqJ01pnZu62lt3GWYNTlbtQ8jMs
JKhbvk1MxOY63bmcobSyqrKx7WVTqbmYusxbd7VVqhxDwZU3d1tZURFxeVVW
8ZG8YCmo4maRViRXJ4qW8sIXKFkvCMcmC8rJmKzgAnTXMfim1OxAMBPyOPCp
JKE7xdTOrXSRch5i53HvIinjKJBfbKiLidey5jazZzXmn2LhXhg1V4UNnEDS
vKydxF8MaC+2KorbfMyXpOtqKLkfyquZdpT8JJJM6XlOIeayFyXmnOPMmpdc
vb1C4RMl4SJJgs5qsi92K2sJfIusMbo3ZwSZwPuGdAzaeMEClduaidiErt7S
qoE3CG4XhQHDkFPSa9yNojH2RMzj1t6Ly9dwtfQ+k5KqTr7RRE0brEJhEzCL
oknCijl5CwqcMvdFQ5qsvUahyZLklyTRko5eG1cTh3RRyMURAl9UpPFwoulJ
1RcypFzuJPOWIkjRmIIO9gIO9u7iyki8EkpEVt0aEIJBIJIIJM7FixvKJJNU
XgyspPuPIiL0Wi5OyBt4Yo6pyHkiDNxqs5gmaGAlCA+WpgPOIJ7nde9VSNIO
U7nMFSkcuafNR108QpIxihVJDUMAsbU6gSDmw8yKvZm2vQKBhRhwiy+3VCMY
HBTvdo7CSi8G7WIZAus2MyZlWnEVSoJ6og6KUxoWB8rZME4HVDM3MV5eSsdb
t4qkVW1O1thU4N3cETSFu5mcU1SzZwvtOpFTowXd05rNxYyxZeXd1piYL0de
LfBhcy5O3eKjmvGTpkHFW2+q1EONgYIEzR2YvJpAzomJ2ybqNm4ujEpS+6Kc
6htoo1MwZJLly6CpPF08nJBkyUlsXGRcbgeqzczDVnrN6QJNC06VrZ+7T2dX
9ZjfC9aDFWIZfsPwP6DuO8/nP0n9R/Qf0nABI/3QjHI6Osg29gXUwnj0x5t1
dE5Tq0RM2Wq9BmVKPYHay1OKVrWta1rWt1+HEgmz8kZ2PJ9Vv2PCCd2MjXpQ
iJ05DtEUVhclIubMPejCDZIO7VFwqE3eZEgp9Cq7zeUdkg1cRDQLVimakXkT
pBiteBwOBwOBwOBwOBwN+bb9Pr9Psq+foih9YWYg9faoVhO/JcHJN5zI3Gal
qlKt5VtBqQHrGDXtO8IHA4HA4HA4HA4HA4FixYsWLFixYscOGmSQGzIW/WW0
IgmpHg8/rnxhZ224EQddUJ21UyhV/cL5uA3UTVZL5mgxUVkzkkIuCnFp2vS1
71cQCs7IKUjFogq3SGDjJlMzMHPKvuAiHLOIeLUJ7kNZYAt1Dx+/IbBOl6hm
2E5OROzPUV3mPeCJypT2QJd2GU4AAbCwfrMyMyw50w9bhWQpGUFByhYFCYy8
FqrrTOXoEh9zWGlmEEBxWYIajoLBjTswBuNRy4UUDgqU5w0jtWzBth1ZdmZn
3McvmVTqsFzEiovUSSSZe4fDBBvbO1W0+F93HIgqLBexk5brZkaojXKlTNVU
a6pzt1lZe7G3lYpcYInIwQm0QY3XlmbakGmhmYCH0VuCpU3RNXj1VgADAXtb
wHhgFLaVjWgSEFQq+OGMWnCIWzpfRlli2NfI64e6p0+1+H4Pleb7fzPuZPg/
ef3m59b61fuJBWmkEFVVVolDQSggFSpVrVKLVVKUpb4OCOifgfckw/UWU/rO
57T6dz55t+LVLMJqWtS+JY2f86P9QGeINC/2JIn8DUyfgJ1A5F6iGo8RvYP9
p+8khhG8k633NG9uE4SnNwdCYwci4JnC7I0HI/wME1HTif1NE4bmHBzYbjU3
DgiyRGSmVDf0q82cQTSchovOEtKXKaKlzuNCn5OpWpWzm2EyRmVyJGjkU7kc
nNM3QUnNL8/kbvB0aZDkJaVRFIJU4IzP2OmyX4DiYLOfJlItvm/JvcU4ni0G
TtS67NGTtLGq7QmxwUwcDczMmWRpveK7QUdEPj0biHc+OPjojg4tIc0unWlI
nAOoWQ0dDQs8kyC65KzdYRNCO13Ev2tx5Jxdom4FkKN5N7NzcXNOW5xUzTRH
ANxxYWPQ3p656psjOarJkR1PEWTqPBvZnJGRHqWWLLyQ5FSShSRPIyNFyFLl
2E3I0bywsBdHE4lhcDbY3H3kDA94GSjVybJ/P0kkkkkkk5JUqBVchDQueUAo
HUB2USlI2TY71MukcVmfaWLOTgLruBMQ6jgpZs0cDcUeCi5223d7zWfaR4lq
8/Dlh1VlU9KvBBOaFGkfdsn4jeYPTKipk/EMMWqVZhl15Wzhc/D7K7MYpJUm
QuwyGWiWKcMgJ+nr36trIBBBn5/Rw78+DxruaNUlSvBodnn2xWdCs21kRKp1
VjeYQeCxYyZbZ21lllLa0HvGJnM3tZx9jnDr9CuxXIb9DBgAQWggAfZN8hNH
ZhwWL0JniZnRZjjadOdKUosss1Zq32OSUzc97zYhj0KFKDo6dTtzb4X9xzdZ
nw/0GT6ThD2T5p1TiXLFKqIqiKxSU7+7t7nB6nXaPbBHyd8+zyFFJI4JH+Ov
WMyj9p8z5nw+yocw8MOKPnJSEhSkkRvUj1z2PhLVkjCQpdY+DGLxsRcPYCdC
JkkVn29cuHPyFiIISR07gt5MkdWfw8kh7aSS/vZEh3vHSTxzeAdZ9RJGVJEU
Sk4Ui3C627pL4DbI+qYepkozSyb5b2KuM4bGIIyyA/apG3VyGUxh77uHoocJ
Ot0k7DD4h7+Jq7FcMkkTWAyWAgfabqiC4y40HUxExXGTAN3UWj4BGcRySZ6c
WVpCCGrltobncvu40PtKZt8b5JeyTTQ/BYshkj+ro0OKiooPcQ6JwOyeAWvZ
X9EGtblHdYnKGAP1xkUi6vlfUGTjlHcjNq8WaWU2E9azm44lECT8UdWOxUqQ
ZTZIVBBdVwXGIJy6QBF8b71SIqV37Nn4OEkLxyLLR5dnxZU4/QsyP930fPdu
fYfW8KfB9LxTCzfm2BKOBn9VWUTFNQGAwA5/vu/31WpDQy1FPSe/7OGnwpeI
vpAqQcocW4TX1Ea46lgxriQL5DVWKQiRtgeeB7KO4Cc0jQIgwEZA8w4GzvN5
uJq4ymgZTlC2TbijXaWw6BhagdZ6BzIHQFBZXNAO76Wb5jxIQYHX0pwfMORW
9unlIYzSAzbPgClYbgW4/oE+Mc92qXQphtbugdeMKxOqySmjxAfKjV69tC5i
x9PDYXHXJe1hmGYZhmGYkJCQKLo5SjzHWIp9TlPkJJJueg1Nr70fn7pd/Bhk
/ej8vIv9WS065ScX8MZNkY6rG6LzBIr1470wFE+ygUAQr3xH+g8h8uOJ0RIA
Mx5HaLsELER4DmMfSHj5TTVaJEd0xP3wdojvIe6QZyKm7yIJJ6SnjLARM9QE
gaDpUIXHCKfNMs8VEyQNH15oMYTBi1xrkPUgimViyiG5qSySN7LCt10+gzfN
un0qsxchRBUFMdjEFiilcMpaPLUhKFhGcY77hfJWn9G+K5uC1Q2R7iT0HiqT
zlCx5DAUWfalMReT+Xb57c+aruzip63yV7apQiwb335aGlhTKAAbCIwX6TSg
T7zsL7fX05N1H7RHynl+Iw38B30MTPYmhGpPXClCQctCCUtyOO8rMEZ5ROwC
EU5dXzeCns4wfpKK8AUi0iTD8LEegkMGIJ5wSoJUEzyYYPaSOunbJXiz6+pJ
ai0ohVCliwoqSilF1pX1xubN1E965SWIU0ZKYtFVChIQHNCHGNgYwPUwWheH
C0nFN6WjVH2CzWJxQo5wIfsWkE1SI3KVInFkmEk5fe1u5uBBGaIivfFiCQKD
MweK9XS/t08rcPyQ2S41JmQkriFh3HtEpsc7CWsUJUEkEqCWglFmhWvGFycY
DaIagkyrz+bMK93rcpJrWizW865L2QwvaRsJJxa8bslJ5lIJ/EpIZqSMNFiS
TIz1btxkF0GBBNCSb475TLngewS+ADiDUta1m4+pWajD3wkJCQkCRW0O84W6
9xPVjq7rJ0+fs5dm1mIvyT5zmNb6TZxxvwxdbMuVS0vvoWqaFnxLkJJoDHsj
lbU1TgFExiapwR7NOcPkYFk3N2vGJigtRbqVUdnLnpgvfkNxUltYtq7yma46
47CS2QwkZr0Osy8ASzEwIdkInmknc7HGRTipM4ZKRe0LhZve+PfF7W+CXwiU
yRoaWJ7IglmSG9GF8hM7KSjwQ9vuTNUs8Kel7zFVC7I3+62oTO8F1B3QQVGu
yB+XbBBLqebIHnJNY/I4bFqIOyi3wuyCiSRxEg6WEyaYHbAvd6DCPga438Nb
cENdaiE+Q6bTFBJ5xt74oms+7FlEZ5Iv6sPXCiTy3wykxDD6V0nzn12iMJRQ
lNDr7idp8OyF1I83l29ux6gzkO0qSXpDrcp1ucLMnbUe8kVJA7DVXYNBUeJg
KEg947FNMCBCEEaVSkpoGgaUPcCpBEaKPjKInvOegNnM9iEMRC/jWe0ECVts
V1gmzP9nq6vqs0p8fP2tfleCgzUktrfDyRUfA+mIsQu6yDEbMtDPFg4VBEag
i+QrPaYzIRk7BJB7IgiFRMS4vRitcL4ZqY9UqQEmBEo0aHnbfppkaZTZFTcb
6lODBRhSNnBMliZFCciZGw1XXI3Kzb0wsZMBRBq2TUt4EDcvjp1jPceoXvIG
r6RwU2HxHJgASAPYwmaXrH5k2dZkPP0cy/TOZ2qOEJued3IeSwyujVTMyAOY
UqlihgzMLsTOIfC0gkwT0HMttIKrYbmc1kXpEiMMNs44x0yrwqPXW9o+EKld
L6oMBkh2BXrwC96Y4EcJQd9mkZbavkFm/f187rtFc4I2h4Hb5FEecr2r5xaJ
mW7PHACW1iehyaFKGoRYsH1aneXMTKG9kespglj2FCbLGGQfOkRwRhlMjdQ0
VTiWPc8jkizDYmROCcHNDROSHFN7QC6TclmzOSJpYJctjg42h3+omkAgISBU
gfYyRgMXcrJxRn5o8PV1Df3eXOqTcdr+Gbv3rkgQt2i5wp3Xzdn62i0YaSZn
zmCfvsIc4ijvOpZiDeCMdwowT9RVkkDOBX75keykKn06S1QulD4yVjd8B72t
EzIvWCmoUQnwk/J6VOHmCFgNJTrFsJjWcGkyVgtciCYPEo05LJ39k5cGsZK/
Qb3n44ILu29TCrlrWb5SdZ5STmRzYJPH23LhSSMSwnrkonsJ7k6OneiuBVkc
C8iBDaf4qTWI8RJlo3GnY0GBNn6Fi0itxKIOThge7YsjThJE9+rKSdjdRJiS
Rt9i6M1nSxg+wmCz+mDQmryAd86Qh6gTfTQYTnXwsm5FTgaofNY6xiK3E1UB
TqSKSbgrCPFeJYv2aU7el5F2XfPeX3snru9+Lbq2zYvS0WEOtqWEwTFbUvLJ
s9RhJsEKTVE2SUjOiMliTjeNW92+gTCLjc0pc2/fUSBDqXdzYp4O4Ewkc6qd
fg0uUj+twFnJaNLJJ5W8KcXZsssd90YzzoWFHWUXKKI2XLhYvmDa0MyLkzY0
QD5fQiUPPUSliLn8Yn39urOcpLOnd4PG7tSmD9pnJdTBnUsL0jpPWo23o3tG
7Q33WkozJtxSJwQsPBRhE2BPK3g7sIVdeJASDBXPsnGOvXI+1/F+hHAnCJ6Z
NYluM9YQTw/EuQQhdgdQB+EOtvcnfWD/Fu+dMyJGx6gSp7z5XH7vvAgUIcge
WTBYMBm4/eBlt99sQkZGV3QDKifj93018frllmwgDprHQbC0hEcLxvIlfyP0
Cm7YXuxcB+1Yp4+7pqwz10F4JzgYFvWwQStFnYpUYiNMWcrWNNM7kowQeSkE
AROikOvFEOwOsCY4kMkI1rthaEN9gITVtwHS7gO+ecnJE9yQj9WPSHAJizOS
w05QKsTSNN9jd58i8lNLSzrQCcSKSJjXtv/Q1srxvxyZv3ZF6uGAcw5CigSS
RhDaCEKg5TdvgcVAF/7BFBOKJ28Orn5dd5NlE2CKJFEIiRsRQdfQc4Q8VfIO
PVJEoZVUcyyy/pYDB9VMoRTd7K3YnjTVTMoonf8vl0B9gm5FjNFLqpSPTvPU
jbNESO6l04ovogQiEikIDmFRE+qwQLDBoINlorrSQIdTBxb3kQoDQ3O/3904
Juia9+bCQAsxU5qdCdI/qYfq0ygyJDJgZqZNCSx7gg2LEXp06IBsOLJhZT6p
ScKaykfSWZuVVuZtUYmkJw7UO4Q8lSikGFHnGu0gxjLBENQaDAS4d9AG3XJC
HMUGgTMEaXrXb9eF/LK6iWI8JIkgg/thaKSCiduMNgyZFoQlyyxhG35GBgkV
JIwn7qzRAGwXrJc5EYwhD4kXzhv1baiRQrbrG9zqIRkYOpKxEKVRoDIUoISH
ncoeZhAq+ZK9ffChbrBhGGmnr8D1IfEgJaAhf+hiJx1eLh7kPQLeCr1WfVKq
qnIu/JVU0XPcybXdFIZv6Xa9z5DqjuaU61DsRSNpJUdFSK0iCXR/nVHh/h84
lh9my9XIAOMBJBRIRUJF63ifLFTJaqZnilI8mOIO85Q1k2VMrP8V+0krtHJg
e1CisCvDJxnEDMqJCHeAoHTBcGZFz09dtuNs/Bv54XY04JbBTvdD0Sh/E2VU
ilHM+bImXhdFou8LE9fNaaPTTY6LO0yTvBfYxZKG/HBgbibtMTCHQ+ZgdaEQ
GEVSMCEIQBlgbw4AZBAccaHVGxNiw/k/Uix63s4OCKVDrPD9LkjHN4eK8PKh
OK6VSlEjweT1l0hitzyyPFJN6Ft01zry3Ys7kPrKSF5jDNGCGb2DykhMSkO0
S8LTdDepaN8LlqVSKoKSU6MI80IIVlDgnipD2/JJJ0PWdXVg9Q7qfKDIQkCE
CEBIHSPIPx/RcsW/WWt8aUQBDOmMKfVhuTA0KWOtdS35cWPvy/Nk3ObDSJKK
RoUpgsKLtyLsl0pZc6hEsrMIYSQIXi6Kar2KeSMrruhTt1jFL2pYiJBqCIlB
SAUgFIBSKtVU4hoa/eUfJMC9lil6lKpwgikkPhLwDIOSNSA4WfEbECijAESV
RktKokCHFhJcySmHA56bmMjgPxJ9L6QUAQ9Idaau/Vp/Nrc3sQb7rhYnFETQ
keqTr+u0M6iXmR7D3lSLLkyAKOYeviihqMiqd86AnPkamF2DdQGjxSsUEe0D
Ihoo9PTANdZamXwFBZi6XGYRsLs457RF0lQR1Sr/FwLIpcCkoMCRYp+RRVUI
Vt9APAWnsO5CoHUI/Ifs1ctyZLKFFKVp4fPUsyCs1L2+y42PSt1Jzdlehbcd
rKoguMlpFV6PSVJNhDoahuIbLPVn1wm/yQ0ipD7CcsJdAvXnRORTqJSBq9o1
NR9jecsSdfcntLQVhEOwyfIfwgcBlC9XAUBlQQ09SrI8U4F/WRA/sIiPI0ZI
MGMGCFdEA1Iq9g2OSHcUGQgnDKKuWw6kIUtPHijbqjeMnNf1I2m81Giw3bi5
LOBZMAAkSMXVTLR5C5SWVHJoXov3B3jBhmwvHY1B/OR6ByKjAjGQKKpNCqqH
8hQtS1V8qJVQBSA4AIBqDYLPSsmaonqcl0YyS7IYRSTqXxJI3bsklIodUJo5
YaQ8ZE0KyuJG6pVyRsDQrECwWlgt/UQbioxGyGzJAxAoWEUJ7D8wbKoAhfID
RwXyE8vJpjlA2AB3QCih/cdEeRYX6WNCYoFqsFK5dTj8hwAsmqbWoCNuOghQ
m7C+CFHqTgL3mFdxybAuOVhq4JqAZK4ZAN75nEKyhCQkYWCKGbYLp2Iq0FQx
hBkXniLIbiKQwymo8xvcNhDy616E5swFG4CNBE+d4Kr1PeueklU0FCLCgUNC
WKWRYNBHaH2kv6aLnVD6C1EAzEpQip4DGgoUUKUlVFmo3TSSSG3yLfgvUUV/
CFDaGMqhoHVIh3oo86A9AQjgjhhFSfQxJ0dkHMnCStwe30heT2rIc2Z3TQzH
GTs6oXHx0qol4vTBUPMlUdlqIXgClhIeIXov5SlTQGC8Bb+9+eBwq8hp79L8
aSUSlKqU9HgHRX39C3hdOHM4Dbt4kMu5MJRL5gnRnvOY31sqAllaiMinQr4E
LHMEHvfN9msGdhud6o8lE5+Z9wqhxfUM005GorxFTxihXi9w/EQcD+tTkg0J
qU+lUHEnPNhMfZMfEEbzKZbAcCVkrmvU0HMfZGl5eV86FzyB7AU2aSewrWZy
ROweHrJqZ9To5Qs9MKS08TxWwp8oz6nd4RMUYvzW5oJkyMlEzChhBKFmFtLN
80VpQi8ZM2aWTvSnd1GRRSUJ1QsWFjNVmDBbHpkOOAkeCqX983E4A0uiQh9Y
uoB/J2cWJKOn2QFsr6x6uFxegUByToBbOcY2QyE33ELCx1+hXJbMzMzM3qBY
EORTt5iZgLzpMzZB6PG4yIpkS2DfDK5knTOMW/QboPUK5HqGT20G/wOZZOc7
IECOTvKB9O58JHdxSPLoiymkbRd4yT1uUk8B6SYHfxge8o+FJOETsSHFxXN7
CjD9PG2qA6CCWSKNCKFCqNIV4YpSwhJFCeaFRVDmGpQruXKFoqu9CQspKkRM
LFEpSKSlIQpmsHtQE691DkQ5sfK0xjJJayhYiHxxSjCKZ+M+LLk2E8dynLpL
VuRkkybcPIznNvybQ0hTBGRVqIrrFL1IMl1kMST1IwR7iQNEKBKSChdFx1iy
LkwVFGNRZaZTnqjCcUb6pFJGjwTsUXJMg1TtqQzRjC6bDHqYMHoolcROYtGQ
b248goMIPez0BNPItUYgOLJYnFexjzgDxD6whYQ+fCMu03JdGxBM0/UI+uCO
k3To+jqnqqSqpWXYrMkCFCiilBCtVb7vvULMtYllCpCCHW9PNwUilUUmxHkH
t0rr3O/uSE1I1GXf6pGn/mky6us4U8EdcI6zDKyLGSiEwpIwqiJaNCQFtSLa
EiMLEGgogmBLBp+PTDyEMBSaJqfm2NiAyeo/EV7BTRVu7mtFShf86m3KTv3P
L3eN6H5nxRvbPW8FpNCLI7Y8DluLI/nGv01vcDQ3msktbJesku/02o2Miayw
JwJx2FyYktT+bFwLkA7NxoRNuIhQY7cWHpdGV+Evud30WOiYHWbrHamxgnsa
2VEVtsaz0uRml5Wua25wuW1CENihoRgLBUguikCgI+gxOkSwWh6CpIYmCRqj
JkPni0UUVVRFCT8VEsCXEp8VIqRRBAQgMBkhsN0RxEJEJFGxD1/sqbUHQmhN
aqaoHBMyplA4JmVMoHBMyplA4JmVMoHBMyplA3dS6Bu6khZCVJCkOJ5jkxhC
k6gSrKRE8pntwAUq4FMnMUbEB9JBewMF04Enk1ZJJhM6SlKU+bYLI3PwS23G
RgO5jjJhdcTjEKU8Ad8qAO/i3KTt0sI/I7yEkjQedN46EKYdmiiAIf4d1Jkj
mVElQbjfi5imDSnwdEOXkrd3xHwaJedsoorhVqqqrijeGhzHVIxeZHE61dh/
OCQTAvM6kpywCNyBKKWxsQSPp2WV7LCj0ZAyE4VjAs7XAYmdBviRpDsgS2zY
JuNWbKPmsky3YOyYLGf9j96zt7fJmkex5o1ZdakWMjqpZnQs8Xcljg9m2TJJ
EYjEiYEKWlhCBA6BOduY4kO6HACIAW38/VhRMpSxd44+aXU1m2wcgJslc6YX
qk0FF81kDcHNFhmlYxJlfRimTW9OwjKGR0L7mZNLTW0/4kXa/TndjGK4U1FJ
FLN6USyybIYJdfVqWMGbMGaLKFFLpRIuWAMDQmduFcrcoajpykpz+C1QQjo7
ubpvMqpfQ3TqWMl7Jqults2JGkTIyONDUUmeCFQss4rbmCWRZvkRdazayAFD
BDXdCHtFJY7jNofjntupex+4yJ4ljAaQUgO7tzIrBw+LYLAkOqgT2EXLQilE
okpRQ4EdUOiYddrNl87YLN2GThAldmSF77rLiUkmQZdjuvJG6uGqhV7JyMMW
mTVQqS/FvRYylfjWynWA9IJRIoEoXTuS8W4yMobN2/Kc0OEXBsTJuzuao0LC
FzosjZe5dSUpO1eOo69u+rmdnBxxjJkcnRY3CRTMssOKknPwRZDNvM5ehKUN
ceng4LnGThzSZC2kcVlRUiNUk233X3Trw3KF+m6qHsdEb80MDsUCaRXmGcch
ta78wJzOjy6Jay/A3OqApJQoVKlChuduRu3xuuSiVw6dSxAig4IKdNxZXRkL
wS/GT0A9egv7ORnkezmKPvUA9YPmZCRXTw+7B3vtwuwoef5nDnUkZypEUj9N
XN0lGhTm7lI1EjaS3f4rd9SpVVnJLj3qL3pmLAp2L9T761XiU7j2qaNUukux
S4upTLfcz5ciPhRc4pFfQVN8WkV6FEvCySzoQSK/M8U6xjyVqSASSScranZ4
EUsid2DziFl6K87HHYXxIIBBIRGNRLm0E9Bi2x3SG0cv0fm3kyI3z2691mKv
dpRuqtCNwSrbNn6GK2uJFIMuJuaFiz2sQiEkmUAOx0vtcLngN82FbW3NFM1p
UqROiGZEjPsR6fIjDDVRCMk3KNVgk0ApCjy8NFvAgo6CjBCCFIDRRRRRQpg+
nmfRzJBSGv5CIIfl+ygFQaqgQFqpSAgNRQACwX45TQSCYRRe4iChkWKq8Ujh
UVytCqVFMGhIELrQjOvorrdBxbtmb+SVJz64PKDsncB5RJPIGkRBiCqTweQs
SM88hmw1UkUy1xGcy2eyUiWSa2uUSR89SSWUgcJB+sdA6A+dCL//F3JFOFCQ
TvN/bw==

--1171294248-886438723-1198775182=:5207--
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

hmm this seems more serious... i just ran into it with chunksize 64KiB and
while just untarring a bunch of linux kernels in parallel... increasing
stripe_cache_size did the trick again.

-dean

On Thu, 27 Dec 2007, dean gaudet wrote:

> hey neil -- remember that raid5 hang which me and only one or two others
> ever experienced and which was hard to reproduce? we were debugging it
> well over a year ago (that box has 400+ day uptime now so at least that
> long ago :) the workaround was to increase stripe_cache_size... i seem to
> have a way to reproduce something which looks much the same.
>
> setup:
>
> - 2.6.24-rc6
> - system has 8GiB RAM but no swap
> - 8x750GB in a raid5 with one spare, chunksize 1024KiB.
> - mkfs.xfs default options
> - mount -o noatime
> - dd if=/dev/zero of=/mnt/foo bs=4k count=2621440
>
> that sequence hangs for me within 10 seconds... and i can unhang / rehang
> it by toggling between stripe_cache_size 256 and 1024. i detect the hang
> by watching "iostat -kx /dev/sd? 5".
>
> i've attached the kernel log where i dumped task and timer state while it
> was hung... note that you'll see at some point i did an xfs mount with
> external journal but it happens with internal journal as well.
>
> looks like it's using the raid456 module and async api.
>
> anyhow let me know if you need more info / have any suggestions.
>
> -dean
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Thu, 27 Dec 2007, dean gaudet wrote:

> hey neil -- remember that raid5 hang which me and only one or two others
> ever experienced and which was hard to reproduce? we were debugging it
> well over a year ago (that box has 400+ day uptime now so at least that
> long ago :) the workaround was to increase stripe_cache_size... i seem to
> have a way to reproduce something which looks much the same.
>
> setup:
>
> - 2.6.24-rc6
> - system has 8GiB RAM but no swap
> - 8x750GB in a raid5 with one spare, chunksize 1024KiB.
> - mkfs.xfs default options
> - mount -o noatime
> - dd if=/dev/zero of=/mnt/foo bs=4k count=2621440
>
> that sequence hangs for me within 10 seconds... and i can unhang / rehang
> it by toggling between stripe_cache_size 256 and 1024. i detect the hang
> by watching "iostat -kx /dev/sd? 5".
>
> i've attached the kernel log where i dumped task and timer state while it
> was hung... note that you'll see at some point i did an xfs mount with
> external journal but it happens with internal journal as well.
>
> looks like it's using the raid456 module and async api.
>
> anyhow let me know if you need more info / have any suggestions.
>
> -dean

With that high of a stripe size the stripe_cache_size needs to be greater
than the default to handle it.

Justin.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.

--1171294248-1768729811-1198800484=:5207
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Thu, 27 Dec 2007, Justin Piszcz wrote:

> With that high of a stripe size the stripe_cache_size needs to be greater than
> the default to handle it.

i'd argue that any deadlock is a bug...

regardless i'm still seeing deadlocks with the default chunk_size of 64k
and stripe_cache_size of 256... in this case it's with a workload which is
untarring 34 copies of the linux kernel at the same time. it's a variant
of doug ledford's memtest, and i've attached it.

-dean
--1171294248-1768729811-1198800484=:5207
Content-Type: TEXT/PLAIN; charset=US-ASCII; name=dma_thrasher
Content-Transfer-Encoding: BASE64
Content-Description:
Content-Disposition: attachment; filename=dma_thrasher

IyEvdXNyL2Jpbi9wZXJsDQoNCiMgQ29weXJpZ2h0IChjKSAyMDA3IGRlYW4g
Z2F1ZGV0IDxkZWFuQGFyY3RpYy5vcmc+DQojDQojIFBlcm1pc3Npb24gaXMg
aGVyZWJ5IGdyYW50ZWQsIGZyZWUgb2YgY2hhcmdlLCB0byBhbnkgcGVyc29u
IG9idGFpbmluZyBhDQojIGNvcHkgb2YgdGhpcyBzb2Z0d2FyZSBhbmQgYXNz
b2NpYXRlZCBkb2N1bWVudGF0aW9uIGZpbGVzICh0aGUgIlNvZnR3YXJlIiks
DQojIHRvIGRlYWwgaW4gdGhlIFNvZnR3YXJlIHdpdGhvdXQgcmVzdHJpY3Rp
b24sIGluY2x1ZGluZyB3aXRob3V0IGxpbWl0YXRpb24NCiMgdGhlIHJpZ2h0
cyB0byB1c2UsIGNvcHksIG1vZGlmeSwgbWVyZ2UsIHB1Ymxpc2gsIGRpc3Ry
aWJ1dGUsIHN1YmxpY2Vuc2UsDQojIGFuZC9vciBzZWxsIGNvcGllcyBvZiB0
aGUgU29mdHdhcmUsIGFuZCB0byBwZXJtaXQgcGVyc29ucyB0byB3aG9tIHRo
ZQ0KIyBTb2Z0d2FyZSBpcyBmdXJuaXNoZWQgdG8gZG8gc28sIHN1YmplY3Qg
dG8gdGhlIGZvbGxvd2luZyBjb25kaXRpb25zOg0KIw0KIyBUaGUgYWJvdmUg
Y29weXJpZ2h0IG5vdGljZSBhbmQgdGhpcyBwZXJtaXNzaW9uIG5vdGljZSBz
aGFsbCBiZSBpbmNsdWRlZA0KIyBpbiBhbGwgY29waWVzIG9yIHN1YnN0YW50
aWFsIHBvcnRpb25zIG9mIHRoZSBTb2Z0d2FyZS4NCiMNCiMgVEhFIFNPRlRX
QVJFIElTIFBST1ZJREVEICJBUyBJUyIsIFdJVEhPVVQgV0FSUkFOVFkgT0Yg
QU5ZIEtJTkQsIEVYUFJFU1MgT1INCiMgSU1QTElFRCwgSU5DTFVESU5HIEJV
VCBOT1QgTElNSVRFRCBUTyBUSEUgV0FSUkFOVElFUyBPRiBNRVJDSEFOVEFC
SUxJVFksDQojIEZJVE5FU1MgRk9SIEEgUEFSVElDVUxBUiBQVVJQT1NFIEFO
RCBOT05JTkZSSU5HRU1FTlQuIElOIE5PIEVWRU5UIFNIQUxMDQojIFRIRSBB
VVRIT1JTIE9SIENPUFlSSUdIVCBIT0xERVJTIEJFIExJQUJMRSBGT1IgQU5Z
IENMQUlNLCBEQU1BR0VTIE9SDQojIE9USEVSIExJQUJJTElUWSwgV0hFVEhF
UiBJTiBBTiBBQ1RJT04gT0YgQ09OVFJBQ1QsIFRPUlQgT1IgT1RIRVJXSVNF
LA0KIyBBUklTSU5HIEZST00sIE9VVCBPRiBPUiBJTiBDT05ORUNUSU9OIFdJ
VEggVEhFIFNPRlRXQVJFIE9SIFRIRSBVU0UgT1INCiMgT1RIRVIgREVBTElO
R1MgSU4gVEhFIFNPRlRXQVJFLg0KDQojIHRoaXMgaWRlYSBzaGFtZWxlc3Ns
eSBzdG9sZW4gZnJvbSBkb3VnIGxlZGZvcmQNCg0KdXNlIHdhcm5pbmdzOw0K
dXNlIHN0cmljdDsNCg0KIyBlbnN1cmUgc3Rkb3V0IGlzIG5vdCBidWZmZXJl
ZA0Kc2VsZWN0KFNURE9VVCk7ICR8ID0gMTsNCg0KbXkgJHVzYWdlID0gInVz
YWdlOiAkMCBsaW51eC50YXIuZ3ogL3BhdGgxIFsvcGF0aDIgLi4uXVxuIjsN
CmRlZmluZWQobXkgJHRhcmJhbGwgPSBzaGlmdCkgb3IgZGllICR1c2FnZTsN
Ci1mICR0YXJiYWxsIG9yIGRpZSAiJHRhcmJhbGwgZG9lcyBub3QgZXhpc3Qg
b3IgaXMgbm90IGEgZmlsZVxuIjsNCg0KbXkgQHBhdGhzID0gQEFSR1Y7DQok
I3BhdGhzID49IDAgb3IgZGllICIkdXNhZ2UiOw0KDQojIGRldGVybWluZSBz
aXplIG9mIHVuY29tcHJlc3NlZCB0YXJiYWxsDQpvcGVuKEdaSVAsICItfCIp
IHx8IGV4ZWMgImd6aXAiLCAiLS1xdWlldCIsICItLWxpc3QiLCAkdGFyYmFs
bDsNCm15ICRsaW5lID0gPEdaSVA+Ow0KbXkgKCR0YXJiYWxsX3NpemUpID0g
JGxpbmUgPX4gbSNeXHMqXGQrXHMqKFxkKykjOw0KZGVmaW5lZCgkdGFyYmFs
bF9zaXplKSBvciBkaWUgInVuZXhwZWN0ZWQgcmVzdWx0IGZyb20gZ3ppcCAt
LXF1aWV0IC0tbGlzdCAkdGFyYmFsbFxuIjsNCmNsb3NlKEdaSVApOw0KDQoj
IGRldGVybWluZSBhbW91bnQgb2YgbWVtb3J5DQpvcGVuKE1FTUlORk8sICI8
L3Byb2MvbWVtaW5mbyIpDQogICAgICAgIG9yIGRpZSAidW5hYmxlIHRvIG9w
ZW4gL3Byb2MvbWVtaW5mbyBmb3IgcmVhZDogJCFcbiI7DQpteSAkdG90YWxf
bWVtOw0Kd2hpbGUgKDxNRU1JTkZPPikgew0KICBpZiAoL15NZW1Ub3RhbDpc
cyooXGQrKVxzKmtCLykgew0KICAgICR0b3RhbF9tZW0gPSAkMTsNCiAgICBs
YXN0Ow0KICB9DQp9DQpkZWZpbmVkKCR0b3RhbF9tZW0pIG9yIGRpZSAiZGlk
IG5vdCBmaW5kIE1lbVRvdGFsIGxpbmUgaW4gL3Byb2MvbWVtaW5mb1xuIjsN
CmNsb3NlKE1FTUlORk8pOw0KJHRvdGFsX21lbSAqPSAxMDI0Ow0KDQpwcmlu
dCAidG90YWwgbWVtb3J5OiAkdG90YWxfbWVtXG4iOw0KcHJpbnQgInVuY29t
cHJlc3NlZCB0YXJiYWxsOiAkdGFyYmFsbF9zaXplXG4iOw0KbXkgJG5yX3Np
bXVsdGFuZW91cyA9IGludCgxLjIgKiAkdG90YWxfbWVtIC8gJHRhcmJhbGxf
c2l6ZSk7DQpwcmludCAibnIgc2ltdWx0YW5lb3VzIHByb2Nlc3NlczogJG5y
X3NpbXVsdGFuZW91c1xuIjsNCg0Kc3ViIHN5c3RlbV9vcl9kaWUgew0KICBt
eSBAYXJncyA9IEBfOw0KICBzeXN0ZW0oQGFyZ3MpOw0KICBpZiAoJD8gPT0g
MSkgew0KICAgIG15ICRtc2cgPSBzcHJpbnRmKCIlcyBmYWlsZWQgdG8gZXhl
YyAlczogJCFcbiIsIHNjYWxhcihsb2NhbHRpbWUpLCAkYXJnc1swXSk7DQog
IH0NCiAgZWxzaWYgKCQ/ICYgMTI3KSB7DQogICAgbXkgJG1zZyA9IHNwcmlu
dGYoIiVzICVzIGRpZWQgd2l0aCBzaWduYWwgJWQsICVzIGNvcmVkdW1wXG4i
LA0KICAgICAgICBzY2FsYXIobG9jYWx0aW1lKSwgJGFyZ3NbMF0sICgkPyAm
IDEyNyksICgkPyAmIDEyOCkgPyAid2l0aCIgOiAid2l0aG91dCIpOw0KICAg
IGRpZSAkbXNnOw0KICB9DQogIGVsc2lmICgoJD8gPj4gOCkgIT0gMCkgew0K
ICAgIG15ICRtc2cgPSBzcHJpbnRmKCIlcyAlcyBleGl0ZWQgd2l0aCBub24t
emVybyBleGl0IGNvZGUgJWRcbiIsDQogICAgICAgIHNjYWxhcihsb2NhbHRp
bWUpLCAkYXJnc1swXSwgJD8gPj4gOCk7DQogICAgZGllICRtc2c7DQogIH0N
Cn0NCg0Kc3ViIHVudGFyKCQpIHsNCiAgbWtkaXIoJF9bMF0pIG9yIGRpZSBs
b2NhbHRpbWUoKS4iIHVuYWJsZSB0byBta2RpcigkX1swXSk6ICQhXG4iOw0K
ICBzeXN0ZW1fb3JfZGllKCJ0YXIiLCAiLXh6ZiIsICR0YXJiYWxsLCAiLUMi
LCAkX1swXSk7DQp9DQoNCnByaW50IGxvY2FsdGltZSgpLiIgdW50YXJyaW5n
IGdvbGRlbiBjb3B5XG4iOw0KbXkgJGdvbGRlbiA9ICRwYXRoc1swXS4iL2Rt
YV90bXAuJCQuZ29sZCI7DQp1bnRhcigkZ29sZGVuKTsNCg0KbXkgJHBhc3Nf
bm8gPSAwOw0Kd2hpbGUgKDEpIHsNCiAgcHJpbnQgbG9jYWx0aW1lKCkuIiBw
YXNzICRwYXNzX25vOiBleHRyYWN0aW5nXG4iOw0KICBteSBAb3V0cHV0czsN
CiAgZm9yZWFjaCBteSAkbiAoMS4uJG5yX3NpbXVsdGFuZW91cykgew0KICAg
ICMgdHJlYXQgcGF0aHMgaW4gYSByb3VuZC1yb2JpbiBtYW5uZXINCiAgICBt
eSAkZGlyID0gc2hpZnQoQHBhdGhzKTsNCiAgICBwdXNoKEBwYXRocywgJGRp
cik7DQoNCiAgICAkZGlyIC49ICIvZG1hX3RtcC4kJC4kbiI7DQogICAgcHVz
aChAb3V0cHV0cywgJGRpcik7DQoNCiAgICBteSAkcGlkID0gZm9yazsNCiAg
ICBkZWZpbmVkKCRwaWQpIG9yIGRpZSBsb2NhbHRpbWUoKS4iIHVuYWJsZSB0
byBmb3JrOiAkIVxuIjsNCiAgICBpZiAoJHBpZCA9PSAwKSB7DQogICAgICB1
bnRhcigkZGlyKTsNCiAgICAgIGV4aXQoMCk7DQogICAgfQ0KICB9DQoNCiAg
IyB3YWl0IGZvciB0aGUgY2hpbGRyZW4NCiAgd2hpbGUgKHdhaXQgIT0gLTEp
IHt9DQoNCiAgcHJpbnQgbG9jYWx0aW1lKCkuIiBwYXNzICRwYXNzX25vOiBk
aWZmaW5nXG4iOw0KICBmb3JlYWNoIG15ICRkaXIgKEBvdXRwdXRzKSB7DQog
ICAgbXkgJHBpZCA9IGZvcms7DQogICAgZGVmaW5lZCgkcGlkKSBvciBkaWUg
bG9jYWx0aW1lKCkuIiB1bmFibGUgdG8gZm9yazogJCFcbiI7DQogICAgaWYg
KCRwaWQgPT0gMCkgew0KICAgICAgc3lzdGVtX29yX2RpZSgiZGlmZiIsICIt
VSIsICIzIiwgIi1yTiIsICRnb2xkZW4sICRkaXIpOw0KICAgICAgc3lzdGVt
X29yX2RpZSgicm0iLCAiLWZyIiwgJGRpcik7DQogICAgICBleGl0KDApOw0K
ICAgIH0NCiAgfQ0KDQogICMgd2FpdCBmb3IgdGhlIGNoaWxkcmVuDQogIHdo
aWxlICh3YWl0ICE9IC0xKSB7fQ0KDQogICsrJHBhc3Nfbm87DQp9DQo=

--1171294248-1768729811-1198800484=:5207--
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
the same 64k chunk array and had raised the stripe_cache_size to 1024...
and got a hang. this time i grabbed stripe_cache_active before bumping
the size again -- it was only 905 active. as i recall the bug we were
debugging a year+ ago the active was at the size when it would hang. so
this is probably something new.

anyhow raising it to 2048 got it unstuck, but i'm guessing i'll be able to
hit that limit too if i try harder :)

btw what units are stripe_cache_size/active in? is the memory consumed
equal to (chunk_size * raid_disks * stripe_cache_size) or (chunk_size *
raid_disks * stripe_cache_active)?

-dean

On Thu, 27 Dec 2007, dean gaudet wrote:

> hmm this seems more serious... i just ran into it with chunksize 64KiB and
> while just untarring a bunch of linux kernels in parallel... increasing
> stripe_cache_size did the trick again.
>
> -dean
>
> On Thu, 27 Dec 2007, dean gaudet wrote:
>
> > hey neil -- remember that raid5 hang which me and only one or two others
> > ever experienced and which was hard to reproduce? we were debugging it
> > well over a year ago (that box has 400+ day uptime now so at least that
> > long ago :) the workaround was to increase stripe_cache_size... i seem to
> > have a way to reproduce something which looks much the same.
> >
> > setup:
> >
> > - 2.6.24-rc6
> > - system has 8GiB RAM but no swap
> > - 8x750GB in a raid5 with one spare, chunksize 1024KiB.
> > - mkfs.xfs default options
> > - mount -o noatime
> > - dd if=/dev/zero of=/mnt/foo bs=4k count=2621440
> >
> > that sequence hangs for me within 10 seconds... and i can unhang / rehang
> > it by toggling between stripe_cache_size 256 and 1024. i detect the hang
> > by watching "iostat -kx /dev/sd? 5".
> >
> > i've attached the kernel log where i dumped task and timer state while it
> > was hung... note that you'll see at some point i did an xfs mount with
> > external journal but it happens with internal journal as well.
> >
> > looks like it's using the raid456 module and async api.
> >
> > anyhow let me know if you need more info / have any suggestions.
> >
> > -dean
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Dec 29, 2007 9:48 AM, dean gaudet wrote:
> hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
> the same 64k chunk array and had raised the stripe_cache_size to 1024...
> and got a hang. this time i grabbed stripe_cache_active before bumping
> the size again -- it was only 905 active. as i recall the bug we were
> debugging a year+ ago the active was at the size when it would hang. so
> this is probably something new.

I believe I am seeing the same issue and am trying to track down
whether XFS is doing something unexpected, i.e. I have not been able
to reproduce the problem with EXT3. MD tries to increase throughput
by letting some stripe work build up in batches. It looks like every
time your system has hung it has been in the 'inactive_blocked' state
i.e. > 3/4 of stripes active. This state should automatically
clear...

>
> anyhow raising it to 2048 got it unstuck, but i'm guessing i'll be able to
> hit that limit too if i try harder :)

Once you hang if 'stripe_cache_size' is increased such that
stripe_cache_active < 3/4 * stripe_cache_size things will start
flowing again.

>
> btw what units are stripe_cache_size/active in? is the memory consumed
> equal to (chunk_size * raid_disks * stripe_cache_size) or (chunk_size *
> raid_disks * stripe_cache_active)?
>

memory_consumed = PAGE_SIZE * raid_disks * stripe_cache_size

>
> -dean
>

--
Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Sat, 29 Dec 2007, Dan Williams wrote:

> On Dec 29, 2007 9:48 AM, dean gaudet wrote:
> > hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
> > the same 64k chunk array and had raised the stripe_cache_size to 1024...
> > and got a hang. this time i grabbed stripe_cache_active before bumping
> > the size again -- it was only 905 active. as i recall the bug we were
> > debugging a year+ ago the active was at the size when it would hang. so
> > this is probably something new.
>
> I believe I am seeing the same issue and am trying to track down
> whether XFS is doing something unexpected, i.e. I have not been able
> to reproduce the problem with EXT3. MD tries to increase throughput
> by letting some stripe work build up in batches. It looks like every
> time your system has hung it has been in the 'inactive_blocked' state
> i.e. > 3/4 of stripes active. This state should automatically
> clear...

cool, glad you can reproduce it :)

i have a bit more data... i'm seeing the same problem on debian's
2.6.22-3-amd64 kernel, so it's not new in 2.6.24.

i'm doing some more isolation but just grabbing kernels i have precompiled
so far -- a 2.6.19.7 kernel doesn't show the problem, and early
indications are a 2.6.21.7 kernel also doesn't have the problem but i'm
giving it longer to show its head.

i'll try a stock 2.6.22 next depending on how the 2.6.21 test goes, just
so we get the debian patches out of the way.

i was tempted to blame async api because it's newish :) but according to
the dmesg output it doesn't appear the 2.6.22-3-amd64 kernel used async
API, and it still hung, so async is probably not to blame.

anyhow the test case i'm using is the dma_thrasher script i attached... it
takes about an hour to give me confidence there's no problems so this will
take a while.

-dean
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Sat, 29 Dec 2007, dean gaudet wrote:

> On Sat, 29 Dec 2007, Dan Williams wrote:
>
>> On Dec 29, 2007 9:48 AM, dean gaudet wrote:
>>> hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
>>> the same 64k chunk array and had raised the stripe_cache_size to 1024...
>>> and got a hang. this time i grabbed stripe_cache_active before bumping
>>> the size again -- it was only 905 active. as i recall the bug we were
>>> debugging a year+ ago the active was at the size when it would hang. so
>>> this is probably something new.
>>
>> I believe I am seeing the same issue and am trying to track down
>> whether XFS is doing something unexpected, i.e. I have not been able
>> to reproduce the problem with EXT3. MD tries to increase throughput
>> by letting some stripe work build up in batches. It looks like every
>> time your system has hung it has been in the 'inactive_blocked' state
>> i.e. > 3/4 of stripes active. This state should automatically
>> clear...
>
> cool, glad you can reproduce it :)
>
> i have a bit more data... i'm seeing the same problem on debian's
> 2.6.22-3-amd64 kernel, so it's not new in 2.6.24.
>
> i'm doing some more isolation but just grabbing kernels i have precompiled
> so far -- a 2.6.19.7 kernel doesn't show the problem, and early
> indications are a 2.6.21.7 kernel also doesn't have the problem but i'm
> giving it longer to show its head.
>
> i'll try a stock 2.6.22 next depending on how the 2.6.21 test goes, just
> so we get the debian patches out of the way.
>
> i was tempted to blame async api because it's newish :) but according to
> the dmesg output it doesn't appear the 2.6.22-3-amd64 kernel used async
> API, and it still hung, so async is probably not to blame.
>
> anyhow the test case i'm using is the dma_thrasher script i attached... it
> takes about an hour to give me confidence there's no problems so this will
> take a while.
>
> -dean
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

Dean,

Curious btw what kind of filesystem size/raid type (5, but defaults
I assume, nothing special right? (right-symmetric vs.
left-symmetric, etc?)/cache size/chunk size(s) are you using/testing with?

The script you sent out earlier, you are able to reproduce it easily with
31 or so kernel tar decompressions?

Justin.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Dec 29, 2007 1:58 PM, dean gaudet wrote:
> On Sat, 29 Dec 2007, Dan Williams wrote:
>
> > On Dec 29, 2007 9:48 AM, dean gaudet wrote:
> > > hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
> > > the same 64k chunk array and had raised the stripe_cache_size to 1024...
> > > and got a hang. this time i grabbed stripe_cache_active before bumping
> > > the size again -- it was only 905 active. as i recall the bug we were
> > > debugging a year+ ago the active was at the size when it would hang. so
> > > this is probably something new.
> >
> > I believe I am seeing the same issue and am trying to track down
> > whether XFS is doing something unexpected, i.e. I have not been able
> > to reproduce the problem with EXT3. MD tries to increase throughput
> > by letting some stripe work build up in batches. It looks like every
> > time your system has hung it has been in the 'inactive_blocked' state
> > i.e. > 3/4 of stripes active. This state should automatically
> > clear...
>
> cool, glad you can reproduce it :)
>
> i have a bit more data... i'm seeing the same problem on debian's
> 2.6.22-3-amd64 kernel, so it's not new in 2.6.24.
>

This is just brainstorming at this point, but it looks like xfs can
submit more requests in the bi_end_io path such that it can lock
itself out of the RAID array. The sequence that concerns me is:

return_io->xfs_buf_end_io->xfs_buf_io_end->xfs_buf_iodone_wo rk->xfs_buf_iorequest->make_request->

I need verify whether this path is actually triggering, but if we are
in an inactive_blocked condition this new request will be put on a
wait queue and we'll never get to the release_stripe() call after
return_io(). It would be interesting to see if this is new XFS
behavior in recent kernels.

--
Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Sat, 29 Dec 2007, Justin Piszcz wrote:

> Curious btw what kind of filesystem size/raid type (5, but defaults I assume,
> nothing special right? (right-symmetric vs. left-symmetric, etc?)/cache
> size/chunk size(s) are you using/testing with?

mdadm --create --level=5 --chunk=64 -n7 -x1 /dev/md2 /dev/sd[a-h]1
mkfs.xfs -f /dev/md2

otherwise defaults

> The script you sent out earlier, you are able to reproduce it easily with 31
> or so kernel tar decompressions?

not sure, the point of the script is to untar more than there is RAM. it
happened with a single rsync running though -- 3.5M indoes from a remote
box. it also happens with the single 10GB dd write... although i've been
using the tar method for testing different kernel revs.

-dean
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Sat, 29 Dec 2007, dean gaudet wrote:

> On Sat, 29 Dec 2007, Justin Piszcz wrote:
>
> > Curious btw what kind of filesystem size/raid type (5, but defaults I assume,
> > nothing special right? (right-symmetric vs. left-symmetric, etc?)/cache
> > size/chunk size(s) are you using/testing with?
>
> mdadm --create --level=5 --chunk=64 -n7 -x1 /dev/md2 /dev/sd[a-h]1
> mkfs.xfs -f /dev/md2
>
> otherwise defaults

hmm i missed a few things, here's exactly how i created the array:

mdadm --create --level=5 --chunk=64 -n7 -x1 --assume-clean /dev/md2 /dev/sd[a-h]1

it's reassembled automagically each reboot, but i do this each reboot:

mkfs.xfs -f /dev/md2
mount -o noatime /dev/md2 /mnt/new
../dma_thrasher linux.tar.gz /mnt/new

the --assume-clean and noatime probably make no difference though...

on the bisection front it looks like it's new behaviour between 2.6.21.7
and 2.6.22.15 (stock kernels now, not debian).

i've got to step out for a while, but i'll go at it again later, probably
with git bisect unless someone has some cherry picked changes to suggest.

-dean
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.

--1171294248-1467159452-1199037483=:20553
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Sat, 29 Dec 2007, Dan Williams wrote:

> On Dec 29, 2007 1:58 PM, dean gaudet wrote:
> > On Sat, 29 Dec 2007, Dan Williams wrote:
> >
> > > On Dec 29, 2007 9:48 AM, dean gaudet wrote:
> > > > hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
> > > > the same 64k chunk array and had raised the stripe_cache_size to 1024...
> > > > and got a hang. this time i grabbed stripe_cache_active before bumping
> > > > the size again -- it was only 905 active. as i recall the bug we were
> > > > debugging a year+ ago the active was at the size when it would hang. so
> > > > this is probably something new.
> > >
> > > I believe I am seeing the same issue and am trying to track down
> > > whether XFS is doing something unexpected, i.e. I have not been able
> > > to reproduce the problem with EXT3. MD tries to increase throughput
> > > by letting some stripe work build up in batches. It looks like every
> > > time your system has hung it has been in the 'inactive_blocked' state
> > > i.e. > 3/4 of stripes active. This state should automatically
> > > clear...
> >
> > cool, glad you can reproduce it :)
> >
> > i have a bit more data... i'm seeing the same problem on debian's
> > 2.6.22-3-amd64 kernel, so it's not new in 2.6.24.
> >
>
> This is just brainstorming at this point, but it looks like xfs can
> submit more requests in the bi_end_io path such that it can lock
> itself out of the RAID array. The sequence that concerns me is:
>
> return_io->xfs_buf_end_io->xfs_buf_io_end->xfs_buf_iodone_wo rk->xfs_buf_iorequest->make_request->
>
> I need verify whether this path is actually triggering, but if we are
> in an inactive_blocked condition this new request will be put on a
> wait queue and we'll never get to the release_stripe() call after
> return_io(). It would be interesting to see if this is new XFS
> behavior in recent kernels.


i have evidence pointing to d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6 .git;a=commit;h=d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1

which was Neil's change in 2.6.22 for deferring generic_make_request
until there's enough stack space for it.

with my git tree sync'd to that commit my test cases fail in under 20
minutes uptime (i rebooted and tested 3x). sync'd to the commit previous
to it i've got 8h of run-time now without the problem.

this isn't definitive of course since it does seem to be timing
dependent, but since all failures have occured much earlier than that
for me so far i think this indicates this change is either the cause of
the problem or exacerbates an existing raid5 problem.

given that this problem looks like a very rare problem i saw with 2.6.18
(raid5+xfs there too) i'm thinking Neil's commit may just exacerbate an
existing problem... not that i have evidence either way.

i've attached a new kernel log with a hang at d89d87965d... and the
reduced config file i was using for the bisect. hopefully the hang
looks the same as what we were seeing at 2.6.24-rc6. let me know.

-dean
--1171294248-1467159452-1199037483=:20553
Content-Type: APPLICATION/octet-stream; name=kern.log.d89d87965d.bz2
Content-Transfer-Encoding: BASE64
Content-Description:
Content-Disposition: attachment; filename=kern.log.d89d87965d.bz2

QlpoOTFBWSZTWdO67yAA48xfgH28fv///3////6////wYEv8LLt93PDhI99u
7HnG+8lwzRIAdCqOp6H3B6Uc6AAborXAg98+eKefdVpvduXWd9t2Re30J1SP
DcbWFa8ujEctza+j1r6+hrT6pSI0BvMr33ur3jdPTbGn3t6t5d7mZeVoF73l
W5F1KWwxxb1FKSqKodHe84BTndysg92OtNc7Ap3ZzY6de29t0aDz7De7u9mt
ZsV0ylyNteygrOxQ3fd98fO06Ho9HdgPe2JhJEECaZBoAIAmaENBDVP0NKe0
jKD1A0PKaNN6gkk0mZEQCaFPVMn6mp6anqNNMGoNGaRkAaDTQaGgBKDKSMUy
DSn6k9QaYg0ADQyAAAAAAAIUkIhGTEjNBU80Kfo0SPUaDNI0HqPUPU8UyAHq
ABNUkjQTTSaTE01T8inimmaBGaQxAZABoAAACJQQJiEyATQAp5DUap+UYozE
mRk0bUYgAND0C/P+GFJ92an92FbYHyBaRBGUCBMnImIHEVIFqUoiAhGEhMgc
GEEyAwxFjBMQFUCXBGVgAIaDECBF/D4/x+PPx/2/J7v1/k/eDWMyxKEF35lT
1JN4xFmEdRL9bctsczB/OpzysfxI7no3JvbpTbtYomX0FsabYuum1+3+1J8Z
ni3jv+jarfBuosty6WzWbdj56pmZmeKqIu/Z6Kf9tTc2YjutfTTTroRpS8am
g0S2qCqi3DjE2bqL63lUVM8K5Q0sqpv/WjThs0zGdqJYVOu9tksurJsurJsu
rJtK0Si6slx6OM49HG8P8PlfJ4bprThyz9VHLcvM140tl1fp7HHf8X1QDQw6
5c+7XfDb4xYRr0yzN63tcrQ9mFzyhlJ08q/TIUNixYnvvxy0ZMiWWjZ0W1RT
MqNt9ukaeV10R0Sr9BWQFm2MY22MY22MY22MY22MY22MY22MY9OhjGNtjGY5
+xfX8rr3RzFcvRwU9Hr1v6duxEfapzDbyOGZgUTRYtC5Bgdgj+Pn2DpUfNw0
DRQ6fcIeJyOIa4V12HKShDPSS85TIPUo/M9JURJVFFGlHYR3U0r0g8G6I8uQ
PVI2SoeAa5mQDmAxNTFSqSw4m7GEJgaWpKFCc8huljRapULgNDQhihgOJgA5
pAMBHNHYimu4bpuUAcjEyAC4DMm4DkYgOu5AkIw1U3OQ4QOEfP45XaTWSmKs
YCMBfFyRGRHvDthA3U3h7nyUSRBBMQRFXjOvvRDLjnhx1ocTslYBIsMFTsFD
VXgqWEMDpj2HLjdEKXENYgZyxgBdMExwkyUDMHWClFx1IAVAlxMgwCBCQjgs
1oUGZmpugC+0JXTzkRbVkWFhIChqZoyOlXhblXUwt0rbtS7MHj8S7VblFAzD
IMw3iiYKBMw2ORkoFuOigb8wqaSKnrV27Lxd9KflvJJ295gJDiQMkNAkHRJB
5fFmDsDkWOy/Pe/XlBhEzUvw4fU9pU5Yqi9wQFaqguWMRGRBKCoLhnjrfENo
tyPZAYQHzgtYMIFSIVg3D38BKDSf3nuE/wGuFNfsI/YADDARrybsu+V8Wlpj
HHCEI6jYIQhDEIQhCLLLNgziIRgRte3LCS2tJJLakk0ti5d/6o+zu7BE8wMi
8uWVr7ydF0VFE1ILLTTTRQU00000000000UFNNNNNNFEuyHyYDQH2GNKggCB
QIJ8vtVe+E91Eyr81gPupx3hq/yFGAB3UnT3GJisMnHjxpIHGHCV3GjYw/Gl
qikxSK10iK6aXq86UZomxLQ9CkXtEVpOkO03aaaPaL10vd5e1YqSZNTgI9UP
nOcT5z5xHSn3uyVuXLhmsjzjZwlnymEGaWDzbu8fRQaueFO3lqox5Z4slRba
5Isc74J+dy9VM+K8WgUCBMFvD8/bwP6CPyPNzV/rwabDiKKQ2LFSsEbeX1fo
+oqCPQCJBGvER77+Zj6K8+ouwI+i5x8iPqhpPtOj0YYREVEXVZVFRudZ6vkg
qqqgqtPMRT+4+WNneaSqYq4VyU3AbixQK44DlwcirjMiu1Qjxbfv+CcogDv3
Ndh44amTMgZkz/141JIAAW01/08UF3Vr1gbQOiGUyP/0OnSzbk8iUIEhqCyp
mcd2ZgIHn0PG+FdPVbRRvKNfxMxz4eyngydhiqcY6MNPcFlXy9DjHgGhEgI0
z4BKcW7ADtAHAcA6wirRlqlvxbDeIlf6vRflyxjbgbDXpBGLBB7Mc8uLbcWL
PH7DvHwxV5zK2iWY6T68U/U2SE7OOrEUO86NI4W1EY7oEiiil3KI/DXqg4az
x5ToHZui3aBbVapIAAXiBQDygV27QHDcdncIva7U47IMAidvJgI4H6+M313D
B1SSEzIwYAC76y+lBO6p0HQoHCMfIENXWDlVSCmj3A24mpaKHQeIQ1SLEm3G
JqGzB2BIcK5IObx73X8fV+HTbYn4f5wI87348aD6lGlpSkg7lpEsh4d3Zw7L
YS5+PDy3IEkgMiVugAO68cCCrFxwoQDZXECatycXR+NFwj7aGl1UN+ELhgW2
ynlEUhEjqAgvTo2QDeyOAfL96PCQZBgbHX0uSJCSRe1rRYy2lxJBGAIbI7q4
YF3mN0LtMQ5zzBDokX2aEzJuY1ATFc/tFsLiVRKMCzInUmXK97ReRRjimalg
6WMolo1aYpBQF4PuGVbbRSrEHWGhqob8IDFyfXiLaBYj2tS0gQ2ywQ4sjZZ3
OpZhNhkSH7YAeBtQO4gn4k4HIp3RPqJMWgaTEuMVQ0k276U/jEsS+t0FW4eI
I9L7Ohy9gBjVnDvvv5sSVICvENlQTZnJsflY4jvvBD0seft0EEG2rmK4BEzM
O2uCQkdeSYjYJGuukwGSlouAIAWiEMGne79rky5ACHZJAqOX0DtpsUDVfExE
U1EEKn00rYPLy+feIyywT0LCErp69pHv/JKcZqTL4cJYG9KvLluc6T16FzM7
zuy47UCgqnCPAVe7y20Tyy+IWHUW/w7eAKnp+IHIO+R0zLSzWD8isyWuLvM9
VIIYEYYsuA+bqCOMFnNs2qoPHUE4ZqBHoNvZ/LR9/4d34fJtFh3RzpmSv+mv
jnqy59ggBiJYk9SXkFIlq6KrgOP2aDccew46+oTt+Cqqqr6d+CIUOCCSEA73
BF2YmYQBPt/hDH/yp2Ir0mI5ooEZc/dTzQjGL4k/Sa9Ph0O+Rg5+pfIGk+V6
Cgq8yvQfEYGEVBFQRHzmGUEVGGGUYYZ92YaKKmZ9Sruqo8uUIoqhFlDKFFiL
KCSaKOZZG3PE+hMDcid1dzfIqqCMczDAwqOgww6DDKqqqjDDAiOZhlVR0GGG
GGG5hm+ZVUbGGEQUbGGb5mZmZm2YXQIOPKaDJx5xlFN8LMVxrJFki5yLUi0l
63s4augy8jnl1epnENH2wD7GFu0/FLrOy4vJDZSl1AQse9CPcjx3sHg0Sljc
z5alazO4lugCHBhj6mmNkvr1aL82mnz2P21uFLBQu8yX06mhNIIx4NxSdu1V
y7c9TXlTqAI68ef66kRWolrZJAACe8zcDpBgp0W4KtqXPeCjfXTNfMcXk0Eq
ATiMsMHezY9YoVH46gdN5c7DmK+mH67/STDcT+/pwa/t5ycbAVMLQTh2aQb4
+rmMf2TlL3dRTnsS1lqQ2t0IcJOSC54YahIPBLUCPRlMf8ves9ecZsuaEDfk
62x6aqM2jsFHE8BE/D6Z4XPA47b+v9+wZfgea7gvwOASSMH7n6aSlCLtQH5/
Pq93ZObxD5wDGvh35KaBMDDenNR6yOfz6SRZkJer6obiYBmBkAlX5u7u7u7u
7u7u7uzdndCEIQ7ZAiQI2CNjrKcCfgcS/llMh8TqMPb86I+dUMUWyqqKqqaq
gwDERwBIKEZBKlFmBCVsGCwpWDBgsLBgwWQwpEEiMiEIwoUyQEEkElUVAMoE
AygQDKBDIQDKBUMJLAywkksREEAygRETEyELDAREAygSMIQjChIwJEykrCQs
IyIRBEEySQEAygQDKBUwEAyIQjKBKypKSsQYpgBlVFFUUURUQVVVVEDVJRRV
LVUFVEFRQ1VVVVVUU1VVRCVVVVVVRVFDIFVUVVNVgmSKEiOChgHfCUgsSnbr
EdWSqmwjIhkgA5lVQUBVVVVVVVVVVVUVVGeMgp+9X5d3aj1nNoWeLw0WRv13
eAmMMbIRWIihCx9zpFPmeAF1APhxqteMISBOfyyUeNvdzJv6wgzZXZ8r5KC6
5OuopyWSr0GJgIYkjb64CsMnk7mJ6sznb/fzaEhImx6p+tft70xYu1ufelXt
D34cmZ3d2Z/y9L+CD5sDNtZJqCwLPb4Z9+mWvW9VWbEkHSJDoKTrQ6RP/REH
8SII+0zooYQBcoumevhWDU3rI1cWzvA8kzbaShi2T8FXQxpm2lvCw+2ygZpz
2VFVVVVXG+/hv9QqPu6/b9g+PqOXWe7p233saB+vC/DoG++mNOk9w968k5wD
o1S8udISWeLDiAPuMJ99Vvwn6RGvnj9Gi5kZx1Ffkiv2iNSbXSK5nRVsRyFc
I+TSe0XWv17dcPGTKluR1Bv33cSfKDhW06xXAnK5atEeKtLVTxI5jkTrJ3I3
iuafXNw9BF0Bzk6J45S5k5/VNWxkS8QarV9fH5gaWgyjJnOolHs8eEw20csO
RICbEgthZ6ObX6D+NRezBl+40GCMM3VdTO5K9kmBEOjViBH5mPQzotq3ClLv
ZJqwTytx3bdVVLXghzJDhR+ui3WwuwA+4beB93xcDQTyNqrd29TwHhBzNAj0
PdXRo8d4iapa5AuoRomfEPxxd3SGYnM8zjiuqD/0cEW1eWoDIjnq3Yd9g7Qj
b2Z+HtYO36vvXk9l5FULvfB3PyWRgkMiZ4733Pe8x2AaAeYPYtbzzA3hPq6a
epOcs0zMzMzcuXLly5cuSSmmZiImaQBZWq0uaGlprbS0ljSkxRybSlq2tKBD
/6E/O30ryw8+X0B64bqr+YIFMeS8FGp0585M1HO0IAyRigXRWqi/FeldamVY
VdLPV4EeqnG8a2I8QeBcRgwXGPCDJXK7WWPLiVXHnv2Gx0VmhYrUcu08ZMqx
K8wdMLWAZFAyOGtciChxEckKm4obrOvfyrtXMjxIuUOA7kdycFXKeRwXKwsr
KuMnAOsVi1NorrWoURCcAFZlEfJgUA9biHj0JnZJqODe7xhq5d5sD/Qt4BMM
KYC/2NtPARF7kJIF+POtNkXbg/378jcqKqiqraPVoMj9GYKWkH2A7ANC2E5A
j7z2YiGKgYXFC/7DPJQUq7gZBMREYWFFBRRRQzEQkJBj7Km5U9Rqe/7CKXvz
v7+V888b3wzveRP8vw6/942BO66numIiPz2JV3dVb8sVAAExYvgBEEP9URGo
Cm3CgIgSXYx2d0IhskY+Qh09le9y4OvgPfgoaCikiFo5kBlMFBQQwEyzGEC6
VIEUsQO7EO+2kxvudEVBCkygoV4GFhP4VosswyTEH0k40q1Pi52WBg9kCS61
bYC1OC5n2fG3z0FlRV7xjNhqxgUIYD0FLcgexJJKSAk6QGhmeNyiKg+BLAfW
vjysYAmACpXU3zXp0as1Uay1rNcFawLgE8CBNOyMOjsI/1IiCJQipSIiAiFp
ZAQG2GeH14CXzrxAvaAQiKgz3e+hVQJHCPDwLh3R2k353pSj3797xbj06Nrf
NXnQY0He06S0yOilFPSipbVw1pStDntppWwLtjwpoI4J0MY/FVnHsDyRUdKp
gYBDRFFEEpo5hoOEDQkqgHgBIkBFCRQEwRak4TJOZLLsEiJSTgAnHVCQQXVQ
WdDBkVbmB0IOAQWSRET9h1nVFVJYFiUveAcGW53AYeSNc/H36FeZ7MAAsS03
fOhWiS+rE+dZ8tgILQCI4BXncSH2OZopxbh1qeYjjvcLHcipbltdVAsFrABh
ALMBAt9bIEa0GyJnWWgdiDYFCCEEwQIUdQAHbCqU2Q4bFhPhtkZmPZEz3Jpz
lMWbSx5ShnvsgoVNE13NNCc7UZoaeW+2pqX2oVSIZow2zO8DcwARTJQTETAx
NCtC25QKQIkOyNnXGZhozcwMEgOuDI2zyK0qca1FKuDaiY3XCn1u8ci/NxEq
whYKdnfA4YW/WnGOkkdYEjWUCchHHJat9VVohjB4zljVm68hVVPAG+5FtfU1
V2q+7l9sgnac6aCT8ORKiKaZXFbTEgtwEG4lAD9+RR6hyQqq7+gPoo9Tu0Mw
DmFaESA9d0+P+Q2SE59FF7HwFCipoXseFxGnYIpyHw8RlhJmZABbEz5f8hAW
AGys0qy+8DMVpYhliVlDCpgliGCqwjKC1q9+VK2liFpU4hwBCSNtsBQdQq6h
WpMjIpYDMwmVSwGQh92tDpRgEJVZFYhYgGZY01FaGzSVpJtpVWiOGrRTKxK2
22rYiwjW+iLMhsCBVhGWR1YKwhCMCEq7axAc0YKEua0aUZGUWEchANaVXBWH
NRtUWYlVVIkeb4/P6O/0d/FO931Q38Yy+DmxtCbjddddddXXVWkktZ6yGP90
Nrv10U7ZS9vNvZuh9eGM3sXAKwHbzrS/Y5S8+Gx4F+63dPdu3Qx/R95ZtOpv
LPBLcllnuxzyI9clwb0Y0hTKRLicAmaQJA6/nnu9aSGmFE+y5HSgIHOIkmQs
sKYDeVtq/4ctr20fyt5craouPN7bOkNgdeMEI/KigDXRoCF71fsUaDpE0xQt
0jjKx7aBTiUUX2KkMMjtnbN7mTpam4R3jUutD7EfKVBYHpqbkMS5RoHdD6Ak
souLWwbGokA66Hf0iudMMEGUErC+BQitzAR5wD10uvQdedWV0z2eqNNVbmqj
JOYIC8BLFOIUzRe5ygRAKnRm3kEjKMCjizzRBUNlcrgHVLmHShYeMDuADeAN
CwnzM6iHK2fZqpW44oQ0hx7ulKmKiwQoVtVNogaqRiWtcsDz7Sc5GSTXdVzo
cKe3ltVT1+JMaqbU9Nx12kmfA1xiK57V6cy6yLkKBIHJxICQMa4dJx6oMNgz
PODjWkATYgmjwURAw7+eOnd8/n7t7I6J5KL4IZrGkmEyK9f378+V6bPYR8bC
PAnBSiCGfjyNhwC4KGqMATmjHMIAg79vb7Trtju3rTV2M2Hw7j+tbWrW7zQk
uQcaUfpa3Ze2j2zdJXYUB19rxi/tDvKxwelbIwkmqMnrQz8uaftjbOgnMN2z
3WqCdUtaSVngcRGqiUxSxpCgowfp9lxKgRiyZ9lpMHBDh1dbrgMXOxyalVFK
AeQUOE48vMWvGYLlQE84oDaIlYtzFoCLxp0F4W6orgaHLG8Q0p2YA5G+yiba
naGSgGF1QP2GEHK/BtSzLPokjjN66ip8BrYlIOcxetHCgYeonVPOyXruIO2z
NYH5iYsg21wcOxISg6Xh6cUOCsghhUCrCSDDCSEUml0/BAFqR0x5yAALk5Y0
AA0UXNIDDOiUO4bor0wFdZHPnfnmdTBaHLBH5lu+TfghwE3qnD09dVqIaRyE
e8Rz9PxphOz3teYY0EOMDpyKKAOTiGHr1VVx5EeMVtWgZMqqyimDKiu1kFXu
qYlVtBgjoovkgHQbKGpnoFPjtips06ddVFOm71tSrxmvby61d3fWWRr82cF7
gYPjN85gzP27UsssBciCKIHnzjnfy1JCV1gZkk9dhSkgABbAQCQf/AOAUMBs
EKZDDCwUhGmhjbX0sabYK7jwiGOGYfpokfuExsME158CtkA4CIbz6hWafdiM
JQD8MJCJSEu4KrcaeO4VeVQcVVrt4ihxjN/fToPFRx9O6UeGtK0JTW+ltdm/
nSlWs0mmkkljUYxPFVNcpERvcxOprug8K6YKOEKGllGsFVtUcIbQN7mZgHHe
hsLMlE+Ox5Fc05ic0RFQpy+ii3LQNOaj9zyyrJyVXQEmc3V2F0UAqC6J5zW3
NvpeeSq62u+6KtbbZ9om5HjN8Op3UVQtz7d9XhzKDN0eRAHIwOdRDCDJAhSQ
MoObZoZXg3DEYxpa1yyd7cc2OdrdlR4BV+FKZfovrMRwwKNOHK1HRYDFSCxC
BCC0gBQ0MEihEoNKAPu6Uxkv7NCtS+n77B1S73HYquCpqovABvEgsZaDEAKg
ogOPEkWQsojPO9BpZJFocQqeym4YXReEVQ+ED+WDZ8glBV/P1UNdnAr71EQN
0yWyj5C9xvulZiOi4lNL5BDyHhrOlCxcUInyQe7ddyvecsVHH589k76HMwNu
EyFXsN8KjQRoYbioBYtXYqukGinXRyO18IiqlLgDqhD9kTE2gKWRCQNut7b7
cEV0pAeg7ihuCBtYoAlLYrqMagYCKMpO8tGQF0i9gIFgdUCF5Ec+24CKYl1F
rnRTrE5GA15tANhsFBQvFXH18A47EUuHVeQ9hlZLCUshYj1xeNW1d2wvf9fu
LfHEdIKGF5Gc2ntRuKZBVEg3Ciq9qjzFDria4Jt29cQDAgBKoG2+c4L6AjgA
lmnQAS4CQZIxRirvoJsddBB4NRQscNWtpGQmCIioYZ4tUEQKHIB1HCNzDA2H
fdRwwI5oG1AzsbpWifXORdGoLIqcBwjdabF+eYN2gyA14OuScxfQQ7NTmdFO
wTnrgoxRsYKLqbA8kKGIQ1OG9aElKDmosc0AwKZBCqbDXYxxfDRrnYwkLeAr
nGgJTuolUHq45wIMGXzbSLRFtNK4ktSLvUdAjIdc2SDAs3LoUNTL7Xy/CEIE
kGMVbR5Vw1dVN6OUXQtTAloISFER8hyII9G9ceNTbG5gKFb3HO5zLZJyNISE
tcclyKDTShoroPBDBTprgTqOLR7TyNcpcRvU59gXH7w0ONQFwbsXJDOxNTHQ
hdN9ch7sKiwqZwkMJMTGBwBpCAMDHO3cI3OlhQ6B1UXtHMMju/k115mGKQ1E
0Nl0nYJilziouwCPUyBflrjgQ6ZMNx5GBkNlroI81Hil1HI1hUaY3UWw46UG
tqd+vKtKK2AeCZpANlP2Al4AEVLqBZbrFLq3FDfG5rlnbXLrjWtcssOYeUvN
+GCSae07O3eiZYmHQpA9Si3UXfjkN1WJwEcb/uwFQDTrlQcGxWvpn0CC20KI
IEJDENmJ405vagPY20J7uNaDYazFU6alVMolqSh8kFmprkRcABiSM9SJ1CLI
aVRrWms7CstbMld06KpN0yXdp1QcTGdBQ9gByAMLvTcyKnI84PKOiM5D2bXV
uGXrOicxsiGpgamkg78NCxaWwsWNTtkkk2tkPKMw5BEYoqV1O+FRySNHSRCL
InbCKIoZPEjrUoTjwUWpDQpaMibuvHtptW9TGtrkWtYVo23quQLg6Oai8z1i
ePsG49iOK7keJHaG6vLsTmTKl2TLYISSTqKhZhJzfnf04cQra1rNlsVbQ13Y
pjjSY4mFpjjCy0yMh0UUxUSKLS1eCjyO+iQs4NS0ZnWl7lrjX0u93tSo6SS5
+VeZgvvw7e+a8gTjv1VgL3KYHUYRCypVW9i0BMbaIiufhrjRDLMzLRJPpc8F
vJxsTADKI5kEsSYAYAxg0O695JIVAxLA10Anq5hvwJbb+sNe/4uAiIjEIyCC
sAIwJgiSgkiIrR0Zy7iCUPB3qCcXpjlRJeBy/YxKOPTrnw7c4ZUvelbZ+G84
10itb7YFvskc2ODSdwzULskUBAi8EA5QRECx2pQBFMAANRGFtcSviiCPoFRc
76ap5tXtBDDtJzAtjrW9Ni3RPH3ZoUVZJoi58vdr18Yrk8K5kw0uo554NAd0
GCEBDMKKr3naZ3gY+Fq52tt5cCmNaSpSVrgU1vwV11UXwV3FDewg2hxBvSEo
DkJCtwjjZHBYKAoW8b0lOhHC5RXHWV2+NQVbk8IrMiuiBfBKWFVtuYiZrfAD
cUIiuPZTl147hyFC2O0m4jQoQyFewUTpkgBbZw5anEumqB1GhVVcMWeO9NsO
OoQ5hAGIrJAMCBqrOOFcxaIrV305L2L+pWltfv5jWgckMdiq8l5JpwOJrieu
X4XJ3b6bVxkVWxpkVp2dgC2sy6K4AN7a2MZClAqwK4eogxDXuZsWlAUzoEg/
QhdYPMTIndke/GKVY4+T3iqEOu7y885RmjMnw8VVRw8KYSBEndm1XUQ49qi5
LkclVmHE6Bp1IUNbbmKdBIlVU7qrxXQjYHl1jP9epMrvejYea4cx5qrnjyry
FDFG1uW2Kimh1R0MNbyvLuSy5EdvzfV+fqRdr0XGvyxTlXC0QNwgUzfWhRBA
/k/H1ibKm8cfVVEMzJi6wJZ7La4j8dVUKKhIFvc0X8K3s+XDTcI0KMmf5nuY
faZtppu5x2j0ua4EqCBfsGQ5x0E9H17/Zsdes6cDmwiOsyj06E/e+3ZHcrjD
4ipTO1UqECBjSmgUD0EvZqwT+AUob0VcAZhUlsi8lNmmOwzDV0wkBMLY5J9U
IZuPNd4+SJmWaZmZnYW5hOMc2A666uj3N9ITQB5wo7FxCQwdNnmPnT2EIGBD
GnrgoHvFhDmCz1Y2wsyYAJA7CNsEy9hYF9pzFyq/3kPZ50DuCB7eIe0PMJLO
hPRbn8aoldulVE6uTgMMkMwH9ya8qB2AZIEtvfbRPu+X8fV+VoI34jBc+Ugk
IMyGjzuuaGG7iZLur1gckmSJQvYaqfBsSJCWGIoPy9duNscPntvdC6F0MUt1
bq3VyV0V0V0quf0n6LrFCvBlKiu737rNiXNUVePD2e+4RIVkoiB/0Kn5bPtn
7+HmopU6UAYe7Lj70sfBa7UhVvKqq/x0fvKVDlTVF9p9J1cSnMBoGPrQLcBn
NZQkJSQomVLMGeKMpTlyTnOKHU432QfNzI5T3RWldVaI63ze21bA9dd7iRxI
5EaWDmq8eMnlOe1bVvEbR1DyFvFdZMQzbaNtU8FXHl5SsB3BrvtB2595Vd4r
aum5OEVyZd+mu8PKK6E2J0nKTmTRFzJxiuEniCxmC7uvT1nqEUDCPSI9PGoU
yBEBMj82cR/g4AcAc8TBPfMTkfURuYQBrR0fWT9dQIClaBJ9FgBj0VRRgX07
F7h/FaIZky36hx0u6Eg/2if5gGoun/b6p8xLIFi0/3G9ESCKecoJt6d++4GA
n+PecM9UbGFt+IamD7CBLCMgvh90A73RFnP1SjiSHesAkd/CJO4oWSCIYj38
uyq+Ql+KnJ6YGCgPgQLgwTfKB1coAlfbB0/4yr/MAgJPBwCoVxwGD4f7ZAMD
QXORN8e39fdv719yoAAyh6gawwR+xYtiW17ZQWAdKqLhkjrGCBQMLky5iuR/
womQZB5IOctNQOfFUtcrRQBDF7pdgHDNj3HeaQjsME+Bjftx6ycTMG06hw0R
sLXuMANhLVhkVDsQ60JOMFCUMShTVJv5sROfHrj5g9uByuIH4wTQGEtIb7Ag
ROscSMosfgbtEhleAddcbiB7PQoFVFK0F/QF+8qDTzGg3zjLsuX6GeAwZgBd
bFWFW+A+RxJPYk64QHPRzlU9f2k1zPAaAammBKm0/1kVO49KCcyL6BgIpmHR
A3E7AwEPoUNAlwSPwi54L7iwWF27wSRSWDmU8ipbNLYj5GhY45oDxSYxysSU
ouZoPxD2SgnaR+LA7sYRBCAlBMIjuCQXmGpR+kxAMCg+JPGIpYTvE1I+AhkR
7CLhpA4Act8Clfib5QxBeIKs2eQQVeswX/qX9DyPZyP8Dy+zAGdi8vB9IBsh
yej7Wsw2ir5h0oqTQ8Pqemr7Rhia+5DmJcJ7hO/AdS9F2PQHl/iWNvgalRH8
z2A06RIPUx7L0KV9DtrW3aYBYbBBUjkmHQ07RFLYxLhUuVD3EDhAAeQdIyDu
CVEDDEp95sUCaZK5pE37hgDMAwExhJHTptMC/UTCgTXtnLpnOR+RqM5p2/ck
c6/gMOQpJD/nP2H2E0CASWhHwcAS5B5/UbA5oTL70qsJ81/6FMlJj3nyPSdT
qfGe8fpPlPzH0oEkgOJStc5znOc7jJBBBBBBBgwZMmTJkyZMmTJkyZJmZmZm
ZmZmZmZnUQIPSfF6fRUQAO+8cEhNwoL5HKDxlREBAKT/N5znnCrvPZ7Dgd3m
ciJvPtD6BQdggQ8xPlHzPRRaKL+gX1gP7BD5B7RfY+6a26z7dpOhH4laST9R
WKV90i3PwVaif0H7udQUKhBDUKjqVQAM0N4H8d2h5J1B6K2iulSvIjvSrzXG
qq6ZcaVZeFXanalpDuDCHCVeleNZeE51xutsLQ2BijgDLC1QubZVXQwFQsAB
F+qNBQ73SH7fL1RXKK41o6yuSHb7HeeKcSOZHQjwMnoVcVXLJxI70t+iWhYd
g1RqCZKLsnxagFQdWIGMuKrrHePCtSqre3aQYotpVen03HncSbxNieN/ARfH
U6ecr2D01OCfIRpVUggVA/qsguIPkZfThJWptKTOEP+HXzAZFA2EeFhXu04/
2f2OHN1AODCnMm1e/gvRWWSb6XS5ato8K0vDi0eUBqo0OJE3EPwNkNQsZ7qN
RijpZPwUamriucm2y7eY1d8WmXPyrndZ5CxFxO4wUcgTtPUCQgJQA2OLwDZD
i70Qmx7PcGYGz6MISIJKRhj2mXPooso0TiHqpQGtVFNx+KQBMxOof0B5gewX
ti9HsJ7QbBTzr0XTIMzv0lnS0CntMcsFHwvv2eYONCaKL5dQ5RUKqLt4IPXJ
bR6qP5uoPzE6qyqqyseEOFCt9jGWuBW4fELJ1L7HgXqEGiilclFP4GZcBXYB
gisEIoty9j0qad2gA8uQZ3/djxIBgI4kQUaVA+WpREqB0CnicQ4gAdNUbe9E
7RTPOsAQAHfDs21yHUkMBgSoBULx7pQKCA0A+LiYLnkNw5i2pwQPwgGIjnlQ
74TWFxVPgAZrYTuMd0UIgd2sqFKzCxo0gmD2aYgw8EEIgBt+3YD8TQU1748t
/WEEaSiRQOdixqgZg3mvu6akHHtSEp9pmmTkkJjoZdaC9vtweOoMMOYobjz8
lBERTBRXJ3UeUqucVyk5VVwI42EZ8Lfhq/LbSaus+qYVaO95Sc61gN4OKuwT
MUNxHgrg3OgdT5lH3vbBoHQcF0TQz6c6B11xpiP2LbnMb4LscdxwtFFrRRSh
UXylOwYOZ7w8J/vwFzdBuwhwAQtQlUYDcf0w/cJ6+8qYKIp4HD7OY0jhA0BI
jx1uFBHzOjQUPKrULAIADk+r+B8zL14bNuqqDoetFKYCKY+muGEzk8t8a+nl
xwyvzPX66rQuvD1FeI2Ft/cCaFxGtlF2w2IO4cRpxFtMFF2IhyFQDjb2wAYF
OGNoLhojOMi3uqy5EdQdIrOfHt5XIi030yWUXkbfnxTLclYA5nEUKIdFF70I
ihE7QXpw336RLGYeD4yQBMNDTzUSG9xAI0CEzIQSMBAGNmwdn0GvVdGcoxec
tMYxjCbiSVQr2AA9NOEm1k6/iIwWZIXgw6yMZJCiBrkK9Si4oprp0L3GuLOk
vilzI3SvPIL1/cVck8Cm1kPaGiVLk8u9V/A8zDZMwgCBUhK/2hkNBAitgnpM
hI4EAfdU0GVGxmFGqAs2K7RfjtIW8qrcrpvTNOAq+4tRVLTsUfNXuOg0TwIo
FSoFBqRqQvi4JD3A+rR49kO7WIZIUIakNSjoXWlPfg7AS2WFNqsw20p9A6xN
itRdPjLgezK6i8gALKrWqNqojUF9ii7qr8VF5+pQeXq7Ev6cN/dwK8J28Lli
t71pX3wphhsqvfY1nbT2qNdSxS6i8FGiOCa5Syi73x0GJ7zWkiOQoSCMA34X
pShKUpTGpQqYqLbXSTBZSTsVXiI43veQxcAKBcyEmIM03gm6hQVrnh3v2guA
bxC1yXhbleikXtmwuexPktXsT3V5/ftZgPWIc1V7Ph6w8lMsXUmI0IPegFdI
hY8F54AeSimWfMMguogzOgDRCYaadc6Crfkota4XwxfVLI3rNBtpEKinVQO0
92GNOzFtmakLtNbVnsgQPeE20J6F83LA959Gi/OAhmRwDqI9ybB3Ir2hQSFI
q4b1wjvWU9ODoR3ut92k2irBHF5RR3MVedZ5g0DxMkKIIGaEKCq1QoIV8inX
YO4ruicMjvr+f91fYo5r2wKKChoIUfeEfkRccATFE3X4erTgtg+Yv0qmoDxL
q4GNd+spOlF/QCZgnDUJpVJGMNcFYCiqCGdDHRrMXthBNQ2ZhicyKKpMCLh2
7a6VWDURqYBtqFdS+BdVr5Fdod9qb9ZlRb2RVlhOxGhM9q5GpbKBqWv4KPmo
p8UDA+D1/W93geTal8P2FlWgXSFz6+PLmGNBXfH3HFfSAMIJCGhUSD9f0XUc
lF17FF2/QXOd5T5Ab1qdFVxUXYDRRTtDs7yAnsZ4lVhDrhl6rgJlgYFjqovY
jwHdA06BxHVRTyDjxdUNgGoesAA4hcQL+oTk+ihgGY20+UmoMJt3r55POXnO
CUdffctr8GDrsj66KBVATAPyyMPFqAgANPV6a9+S+BXc83I1IodoJuotRKBU
UCk62olZrHH4/Vrt21p3UgjjODMOfNhLk2rlguhq3cnAlglQCAilECO4Z20y
OdTwNs7EpJ4hgOu4dyj2OJ0xxHxUXI3alDYXkVAwSDmPd1VWp2AgYcsxHxB7
RQ4nXtSebNVaFVUz9nrc1LoRx9fCXK/BY6yxQyp8MVbc4PCPVKPhC25VjX2T
3ZSr6ARuMtdAeMoW2pDuI7hwOleaF7YZKFj1w5wwtesKEGuY3xLFFsLaAGUH
Z36rUSphhEVq8P5oAgAPaVOZsEogah6vtQLp/XohhqqhY7enHmvTgJlhhBos
RcxcyNFJyO3kgJ6IE/aFIeQksJ2fYD9PZfYNTrxOR3Q7IsLqgSB4nIz0V0MU
zxAzqGCj/Xd2HUs6A5owV4EBqc21a98PExOX0UKi4F9E+JZZfeqZUUAZVX7f
cfG4F1hAwUCKlFAl8Q8El9rkzxd90ZZ5GcEBR4oO9IS5ByWRhDMBmMysxcA5
L5uVPObvQL7MyMtck6zoVRXIeSjp0Mkpn99NCcHEp1xGw/ZMONpGZmyLAbSs
NJe7NQaqQp11pRIRFHZ4jW6V3wcMuXK5a71ojw1xB4A32V8rzFIaiGljJpBm
6i+oaZniqvkFPbKhBAMKgAJYB5OzZoolfj0XjRhG0rAQSHSSo0cILoNgEAU8
ASJSfymNGbUxpP8HItA6nVVqiACnd4Z1D0LBj48ahYV1RUH/JZgCdeXDbpfM
mKLkAIDAEUiCqViXpXy3wVbT5ITLvWgNYvsaqq/0mFQOsPur36igKn1FCCID
BGACJFGFe5PTPLUAAhdE9IqSIEiHG4e2q1ECcg4s+8iFlcSIvgosAD7ukoZC
4e/ColXGqah+3emFgQkan5wUD9YQDnzzmCi1FDUzLC0IfUv33yBvZVXqNntT
ZHBHuF86x5UJcGHtgKsCQkfLuibwptLhsbikFIp+mNQkyCQi4LIhVAii/OoD
UILAhQRDmjttVXPC4kQrzFC6i2saQkjBxP+7SuMQo3RyDvEotlF0EFfl0C9P
IxWq5ZdfUq/qIJSGiWxoiHUXuETlV9UkkeHONj+mSGhUUp2Hgacguchf3nKy
NU3FikReDTiq/Zj/BCh6+QeLo71KCJSl/qXXnMwtJUAgYFxCxBrSqGKWcv1O
YNDG9DECdeuXmTN+nIbukgaUGVv3ShFYrNZeHsvPF2cS3Hih/EMCRSH70Kgn
CrzKhxGljqfS9YB0B4lgkVgAD2AYYFFAaHgusRu+GdInl8QnniFtD9F+SW2x
774eFO1Eh7zur+0ngQVzTQ8HzSqDdFfDuFb+7K88cbULBCD7lG1So3T0O5BQ
5KlEaGS+CYBhdIbq5shSkIR8B6UAQBuYQOA9xENPDAHyIQgk0yD8qh+2xY/G
5V2/DoVVT69i1yhWg1x2+ff8e92vC3DR1FQ7gBCqDKoIADShTSp+mp+oB+Wc
gAOacOhvp6JnzZA/D0gxtEU+ZRSXDzmtk2w2smXNjgXZRTQxGKQKSt+OpCXj
cI2MG4HRfgqAAOgV4W/nzkPkOeZ60axsO7Z0qKpYCqqxBN5oaVMpCS1w9RdP
iPtK6nS9D11Gpk6dKkgmFjaiIOrNfiUFU5BBFE/oZzu94CS1ledfh+gHuHfA
ywWcN8EDvUbosEJPf3XSbfoqPDkKG73nTgL0HIEBYMoCJ49aTQGRgNqOqSJb
PJ2KC1awfCKnlJFBYAZ6qehinoBXlAO5H29algN8aCnZ3g09CAf4RQ0B27Ik
GLYpdKvO9tTdV6d0t/6ODVdvPa7oaLwnlCrtdZy1TlDhMrf/vem3vuOm0GJi
Q2sJdHS4/QjuQIyG1EkAzh/OURw29UWQLkThFbRkVyB9K4pe9Xja4A4fcqq3
isQ+iNQuPbrohAr5kXcBNjjgfqACo7KAEzzpQSq4CJzcwM/sxDckJSIB5CIA
DbLYcwwQ29HfgqHQIx95D8pPEIdSrCIgWMtPM3FdAALBjnaqGhFPcCmhgDLA
oOYJdQpwbXypgRkJULg1sVHcSUVgWtTyFcEDFAguV7pq8TxwqGuFZYrc+K+e
9SuMMKPot74auqwKaAQP25MEgUD1R1Bw8wAOwfYfmNTIST+oElZYkNRTiAnf
ooL/qK7A5BPGwZgXj8YHz6pHDQSnAHVmFykVSyjpBFkqQoPp7gQfSAgpscYE
wzF+P53KaYPE5gockU3HmacOJ8CaRxBKq0iuDR9f4DfxMswMRV0O0+ScHxSY
7GIoGor84D6kDMsBGBG22EcyZAwcPgVFgIimDgztULcDUPSHDuD+JXueGY+Q
Swq2DQrrx0Wh60IPQOhawAB7h6iPGdVKx6abkvaQ9xDCwsgBiBFxiVbGYoUB
NQEwAPUHJYgwLk/mOFPt9QqqP78jmShuILjai00RWBqmoV9e1+0MFpl+LQUf
yRDA6nZ49TICgEjVogCIF4nBwcxRYXsKgF0zVLGTxciQPy+SnLwzV/HYGh2v
MHsFeZxRPYw98DedAHaOjDvAA5GIx8BOBClURA+KygKqUmp6wRwcjw470N3Z
aNzCq1cfpIuke5ZTLFeWR2llqiG1wfiZCNjc7ThuY6rQNxceGCFUS0jKQEzF
aFbFSo2qCeo2BFIoxadUBP6aYqaC5yCQA/LqfyGCHfADQcIIG0GbLK1ZfDqY
0RwHsZql+7Wte01xqo3qGQ8/uGgdIvcH3hKoHyLCYaUctDD9wCaIC8SBQOu4
h+4vzI6gIADJkThU/Mi0RW+QLSQVAAdTY+9kJAjmgW8fLA69HIEyC/XD/rEt
tpsCweMoRAqSBWYSAN8VNoCrmSY7J0oEAIzkMbQdnBMZavfCgfsSyIh7zwEO
CLhFoZ0Y0Cgh/6sMG6lwEUglKDVKwlSmlPgYcRhKAmhMFrpKUfjaw6EXb4KL
RYovBGiAqA2A97cG9BE162gzs8gMipyCYgzLqykKs3+Zu+9ek+iuVErJYVGV
lQt7ElvJhkVtSq1RTImINEwnyXB5mqr46vGr2y3uEmhFPmj+i/KhrDVpDWGr
SGsNWkNYatIaw1aQ1hq0hZYhZUYhZUYh1nbwjRNgSgoZo9x+nK6hASwBACBg
5Uu+tiPS/O2tBXetL49cVtWPV2RWGRRTqKdr28hcgDwDeQkJDvDMl0Vo0zVA
AH6+dOQJG4DsM7PE2uYCn6hinFhmGhAs0LUBBuCJhERmahUIQTYuqK2E/GmF
UJwoQxCsxQJsqDMsRWh7S9aiZbbdhZE+R9QPuqJy0ReNRGgoQF+3ClKHkkqo
px36WPlOnmnnMbEGWo95KlJQauIJRuEIHXcmNzHEbUb68MIdAS7yWQksKuZU
gjSdrvWtW1YtqUSY1SEqIoYhcKqPARuJgbVNkyI2CawMdTSblTGDkUBAyBxF
aFkicBAL4wEXLICHuE5gtypCwAFAwD0KAn5A5IGhtATDxYCeAJrKkAlRlEhR
kCNATYORYx40pUysaKZKkBUwD8kaUyoq3QNjrRQ1wg5wacW5oQCKZq1VPvKn
kVBdAGq46exLX6BiFHFJyOhRNws6S+FoLQWadDeBm83lHQAAYCSgLgqpEDDC
Wj+EEGUqBFg5kJxJJkkBNjN2FdeIKDkUzsgVyKIGM0NqI0pfYD0Ejw2DbP6k
uXzSijcGBsmwYBwdtRKuJ68h+gvoiinaSH8d/vuUHFPT9RprAEPkJ+uI99jE
hxMhADMCnuVeqviAtWyz7QHuLdSnpU4cAT3x8UDT2EDJKKeNFGpDxYJEPW12
UOyLmBez5gSCBCkQ8w71V8gyadiZh9/L62Evzr3VJAIBVRYDwEaK4xtK2rms
tWiLYltrhsvUiu3BIOAqG5Hl8IE7g7ug45DCWjEaAAHQKcxzQFoFwNfjPbsK
R3/giq/vfXqVQ1rSRGtZoiTWJRbViQ/VslJeupUrtUrKVZYqrDEBksQV0qVV
MhcZD8dees4hQocoeLEKBS1iFmwUTGPeYitg8irVRT2QBHNAPiOwAesR7lOQ
Df/i7kinChIad13kAA==

--1171294248-1467159452-1199037483=:20553
Content-Type: APPLICATION/octet-stream; name=config-2.6.21-b1.bz2
Content-Transfer-Encoding: BASE64
Content-Description:
Content-Disposition: attachment; filename=config-2.6.21-b1.bz2

QlpoOTFBWSZTWQl6ozQAB3rfgGAQXOf/9z////C////wYCHcAAA99uPRIPO+
4ADylfFXtk7e48AAnVCIFtp6rg6sos2LsYXDVTW9tVTobGDRtgyBe1Lbevee
aah6Avry++bZ9Ml7O7Xw0EAJoACAiNE0bUmyRkabIyjQaaANNBAETJiJpU9I
9Q0aaAAAAAAA00FMhPVNppTfqp+kmj2qBoAAMnqPUAGgAJNJIQmQjSeqbTJp
GQDQDQADQaDQA0IoaYU0/UzU0aQxMQ2pgjEwTINqDINAkRCaACI01NAU9TUP
UeoDagDQAGgHf8z/36RF/rebDoyT1Ww2QqCjvvihpBBJW7tHClYW2tdkWv0w
73rNIeOGyX58+l1EVfl2M6J/dNuC0Wf2/lcOGs+buCBMwNdxamKz/Lmc2uDW
qLFMpfjTxAyPb0+JmpWLU0zFcymJWNpKwty3AUFFgiT5NFMTWOYRYiC1AWW2
REitpilq3LUqW2ZluNmMuUtpjhlgmVzMWVWVkxMM1hC6o2paiunEFzG4PlbD
EKqSukmMLhRiYlLYGJCW5hmXMoIxGuZQuWFYYkJWfUcGApqlrIst1tqa0XSG
ZSsqWUsWuOQkFJA+JgGlNRdMtLQS22qTLjiuDktrBy1LafJjrDVdE3+PDF1v
uZjV0q0tpViwttlKVuZjcxrUytsRRQnF9/fWKxNt8cCplFzFGnDCuMV7qFym
yLapWC7XTpDSGJhVFpjEctDBMXGwxwcy1EYKOZirbUsWlaOWo22VNOJre7OX
WIwcLiYc9YqxBNazLbMXLW5c2ul7U6sNmoFmFGhlPMokKOMqb7L8+GZfaMFV
QPDLTlmoxDeDA4HLehCAA2tH9Rv4lB03brtRX1aj8rQH4O39sf3P82LknsQP
ffWzXg2V26fH8uoTkw9zwiiycPe80SMvxMgvfLvBjS5HPFGmOptMBvDobQif
5XLd3QYO+d++Ot2SB3ZeeE/jh+kWSFu1sdmaGmNT/fGXnrDljFbJw1h0+P58
fLw5Jj1CfL5ou33e8bM7Sm+NPN9h0/NZM+k/lL77bGhjPODrGuU4bE7451fA
nX5rbydMcLv+PPJOsMRW9vE17/q3eXJZoN/gt+Hw1zf/R4hIWUuVbZaND0Sx
MRNiaZJpGIs03Bz92dLrk2z2dwMfr6Q5HldfJg2lfpXe1j6UdE2Bh6Sg5+7X
zxG+16qVUsdfs1dE5orb3LprylS8ZnzrDJunSODXyPG+Q/H28t7X86SGUxrm
ZM9pBa3EeWoe4NiTKetDZXEGjRhtvoeLcdo8WpPaM3l7r30V8s3YTIKV/7bQ
ximb8nXWuMrrluhi77pzfDDHRIWQfpYzk+oqndtz3AaW4yRXdWbx1SlmXeNk
25b22hmLKNXnGJnxlCqMwpczuzQYvWxYU2ixRM5K8S0VIz734Mum9VRJarlN
uO8eOJ4swizKCBp2otMXO6fP93r+jLy9aj2Zu+oEAAAPqT3+33vd0YMp0iKO
gCAAAD+spOH2GIm+SW9kAcdHdXsa/BzPtVXe6dtf2/nkY3AtoAIAgWp7zb4I
QAUKhWjNPG21jEdspHUfEG7pFy59cVSVtGOrSEDMPYK0cWM1SN12OHNeFoa0
KHO3UsLPIk6RY0B3ZM5UFza77FtqInCGM91ulXh5SGByU3yd4guc3DSFzQrI
rqr3sxcoNHAaclk765b2tJvOn9n4+/9fxb7/1DF8AJLgNLNf08HMA4B4GAZ7
i8SDv8tadLIjKcSkGejj863XpYXIICcRiyXxw/F0XX4WnIfYyZPd398eoIMB
mHoLbcm3GASdbX2gR+jjW43YhN+3DBUm34YODMrjmPyZIx4mQ+ItxDJpgkxp
NdB8ob7GD6LExsfq1fUH2Qdq/DQpi8pirAgDxcMRqLDvPnlWgvaWIZ6YRo4V
LdBbRVHKVtGOHGcypuhAvLPQuc0JwiQnAaX5VyFp3YqKHrMIsc2d4Je/Jmw5
2AIRg6WbG5ufWUruBmYBDAzlxY+9xUq1r4oFeoZ8s2MvhDsgJcphB1my0D8G
gF6V69GW2GQIGo2tW8Kt56OdXqwLo9oUzPxlOTKHwnnp9rR5elh5b8deTFdX
QMB+w0MjYfCj0GzOnznnzdbZGaSa29qWeEPKFHdOOcrvq8Ivw9fenrsxWkWN
bfdfvOUdJ65N4OLMsoyjUXLFi2JvRq6ahUlva7xks2QtVNJDxu8XRqa51pwm
+HKAiYQ0jZFmMWs4vq+ulLqt2tevzGjlmywae700+z3l8NeNznTOoo1ziDc6
zdq+1i+DxJTpXKZrHF7cZl79s/IcR7FrdLrl54b5loqxrAlKNbtLao0rR3ab
MqbvqsV3HGEW3HfO/LDGiXc8boCll1ZPiyrdeeGIzW2RurbWuJ+zV3mtq0Jd
RSUl65r5KquK7Xxht2LSlxPFcxaijxm2KvLWlaz9Uw+lV7bDV+rU0WkpWIsv
pIYQGtqcJxOgfsawPOd+0Gzp6jEigVQpoJLVrAZrUqTYl1syvoq7SRHLa/LQ
N3QgieZgiRVU4+RVNhUWfoms87NYIPb7xzeEhdWstQ+/kttBjpTDQoSGdo70
hGHcw65xO0YIwYrApN/vjl4KBWIMNxjrata1doI+Kw6mhcE2DyN6CnmmzVWt
trdZvP6KeOyNYkrJMlLix3w4BLSmGyxGlVKEpC6vA43ZgsEB41PTGHEZMhNr
WR6Ya5EGwUoQjcBg+BtzI7lauLpExA2vopS14F4NUQg7bBtu/OOj+Xc6kMQr
Ba1lY0oKrFUfR3ba7k4jh2Cpcnesimi7c+ao5tCgQ1rr0dZyxrSxcsZYAYgZ
KdWCuTJ1FnCZeGEFigClLqqSIcglAJAhSASAft966+FPrvp48yHHbHVeu7js
o6gLLtTysP2Q9K4EYxqwf2IozRAwSDJ9fRLHSj+PBgUOYICWd6oUJgh9eIsw
wWLlXXIp/x1A91+0fBDf1B5SCDODFMWQWCeSwGSOwJSMSX5eLCaY4xK15SB0
IUmVVWhbETjDDXqGp9K/QDlISaMzDjoWQlHiiW7YVCG4BMMKo+cOmCapYdCm
LzRNO012VbFCLqhbzzW9MDA+lpYEkLll5B3QuQ3Y2nolw0psx4fSZO28ie9r
yDahsY0m2iIQfLA5DqlbiVUYvUpScTIr4oYWjvvy7d+37fMEFiyCgwRiLEQR
FRRViiIqkVYsGIKgoKqDGDBERGRYxQYqMEVjFRFRURViLEVZIrEVQBVRREYq
IrFFWKoIqqigwWKjEYIoRioMERirFVFiigqIooiKopGCCxWRkVBBVEVUWCjE
RIsPY1SIxYCCDFYosFZFFgioqxVgiwiiKogyQGCgiIqxBRQYqO1CxViILEVV
UURFSMYCkWRFjEUYiMFIKAiwUGMiKIkYMjBQREUREEFjBQVUSIxkBUSKQoHh
oPUHwwu8GjU/HAsklQxYEh2dJR5Zkths1lIFs54KgJOhTSF2q8YUjGqVhdh7
yrCkFR3vdhPKyD4ANg6QRKJSpeWIkz1m7p9+FkwHP1whnKevmntrXsvWlcoj
8ZtiyHuzhWWmMEEhIKR3vv3W1u3is7ynaNXU9hLSXGORN1ISBMi1HNrcKpx3
gtQxK5GLsMhwkA2+c4SkQ6QEJM4zyU1q3BiPb4c9Vr2YI/QxAS0mNJXxG00z
L4zFN2t1MGzRriRS6jxhQldllMmDMRqAaDcZe/UwY1OaB3UXPnwFVon5wVMz
OTUzpwwg+fOzazejopZgvR2EIFjOWmNFmB4Okd0RI6HFalc60UEoRq9pXBh6
GofWus1tXgDLxWp+aTyCSx9YTXN9X9HD5CqBfy/Z45pFsxVphc5ESxqznXz6
6Rfo1dboA8fWm4+K87KZo/wywI40dHghwr8jc3iZn2yS72LhVK9xso9T1iJi
1AM+4GyVwzZnGaRESu6j9Ft8RXcRLiMx2h087Y4elK3jc74erC+ebRKuuQop
VGixSiPTB5DqAUWs63Q0WK9sTZM4faVFYd6SUT2rqdOmUGniuVuO3bGa7ds+
1LEuwguAaIhLHH4EYa1L+16yjWrs5v1pxQ7zDbcLHThpDgMsxCMoCmKd4PB5
IGqerixHuqO5nDt6KHCA5QqVrJ6+Ccp2IECKKGXNFdr9oMEiJSNs+2RfL1Ji
+aivrcSrANWkaBfoli9adQqIs2a9lpZlhTQ0eKTf0E6tjv2+e0EtNp+X6Mlg
ZP2r8/HtXlh2SQhJVOved6k1IUv9zW0mhNdalNiLCzGe6BpBVoS4zUrDBRaf
NdJzBbGFB4Rw2pfOOV+Fnd/bHC6GAgaE3SinuvO4SglTxTbnt8A+bUGTdsZl
gOTCMDQ2lkYe00TjEqiqEYVx3OdEd+16la7fnAABrqhRpdg1WL+tOaItZsMp
XrlVVdcrABkxDaESc3ECIgExCBUgLJILJJBQhKklYCw9dsiIEigBFAAWCgBF
IpIiKskD2MklZITzDjqPTerMWvNl6HROuiiMJ4PL1csOSiyTjmWYJyOmWdtb
a1lpvGyBkPCqiBoIrBDiAQApRZjzLR0EIykdjHMUdpKd+t2cg30n2xvOnbu5
cipaXqlMa7s486id3PM7zrUAFAxJlkTWZAfwFIxA7C+G7gYcj3hLaxoLLPFK
BSc6pDZtm2l669HAlmmkYZYBGtqpZT66WL00ULbkh+uC7URyqghCSZboKEmp
RdWNaBeJjRAQlztBafI5G+0EiV0CKRAM+3JijtrK7sw4cIzZHKZy0C9Wmzld
+vTXGfs7ZaTOKCDYYNtDgXd0UtUJ4xuywvFRvVyaxwynUJYNNNHl9tuRwXol
R8b+jRnWpUgIDSfbFUqh1WeQF2WAKPcTDdbIQ8Vn3Uhor4lCeQ8KDlNsPENI
bddGHvJEWKrHuLVVmrVPeSd+bVwwu1mBoZV0ygox9MOh4d1I63IHmz1ysWo3
tnDfjGydOzy3K4JjBgsXfi1SfLaOB2pgspUsuenxACiGWgM1QIACwY9qNcja
fRIB4KFIUNurgPcb8mI8tZ60MlDuXV2R2LsG+Gg3fgad/J1SBeQ0QjCcYEZI
7LpafSbQuxOaQENuq5WFLihXx2q/F523M7UKOIdVqEN4+Ewwx4PiuZH9dQgv
3rLPIyrjcCiK2GIFG9qERACk41kFHks6qTMiAxKKo1bUQMZVJB9MK6BdVJbI
XqpUWOOBizwaywJJoLlABfsgpPprBehF9xCBLlMFQdtqoLeZUpIvnMy8lcwd
n4jwORLQmpky9QOhsenPwvB3eRfSUXKR3bz7WKLBmGjFFtrXoFxKIzalQxqB
sDMtCnHT31iqkzNomcUXYMZP2ku+Lb2D6jXqw4avlDveW0qA2bEdmI+NyyJE
Fun2veDksaXDC9N32vCKFgyjkTwnuocIjA9Ed2U+r0Mbmhko8lyBWCj3w4ut
tkZnhPiYb8GmV4Xy92ecHW9m4hu6hmeEPR6bORdoIZduNuNaS61cExyibNXa
+3TDsZuAeo4cN3agWIpBVQYal2skISSwyEQqcuW2tLWO1WGhIsGNRQlEFmBr
pY3QxCdlU9Gi6DJh5zte6YYwwl30FybuXYQ2Jsme3z+HjjkhRfQ73opXgk3S
zv3rEMLM2e3OfeM7WOCjOAHB5usoM5NnkwqAmhk9996r5iy2Tv5bHwHU2Cme
ZZixp4MSEYASFm3fbzF8rEEGxJJANELWc5gtFPjQo1OK+VON8vmmISElw8WB
AJITsyBTO9ntOwMpkZAV7IO0XKHTVukEsAGnRiJdNYdGOL1RwVVkZ1MvCxXO
zWVUYrfcZxJOCdPU30cdinLie42DuYCnmwqsFBKqlCChIGyxbVUjbORlRQo0
1sdaFTI4VabiMX8e+eEGGuc1xcHZR3g3bTARiFWc9UiY1dGfVwxIJhRywNbF
Vo0m5wVzPTDl8PlDz5G0vKW630JSOCdI4gmWMaza8ZYFl2zuSxHFpKyU2B57
DmHnBS6EnRD0ppxj6M3wU/CPRlu15aT+Wep4SreditMRh2YQ92Cwah2X4Gk8
3y6PN8yUzmKZto0MkCm1Ve6FmvYowFGJO5HpRd5tML7xW8UA4GFIfskR9/CP
kMNxNlNoWxTDOxwzywniIwgrpiRJ6WhmZUuyoGNDZQL0j3XsQlU9Lq60IBk2
R4MJGmSbBl7aA3AB2rxYoli6uzEQZSHLTJGFldSw23gqY1xvnBDjSMIIj1kE
Q0Le6hGKpCWC91fE+kEu+uBpAEBk84Sz2IShqMPxKUyVT0n0qiO3y+FBm+re
6VmtXPpMMGYOyaOhQzEiCUBlh34aPa6MvhJIb4Oh7vaG+LY9J/Beb2HmmY4I
mU2KakSbstTOEGSgp6vV5rTiIgixrRQKkWeK9r3TcigKAtkCiLgV9eguPatZ
5m4EgI0YOZGhXztlg6d1+m2Gwcdsdp5EyoVn2HDbEGFCikaImVGGykOm026B
4heA/42rKoghoAbMAmOz2ysUNSiO9OdTjjJHJCICVn19qmNkhQscoAh7qhUo
oXDWgxdopXVJsOC/5XDPPcGvQoLNHtBkIWTSDOW9GyISLufvCPLRVh5Acawc
nEBiM8MWeEHrjfWjZze2h9tK+Kc8uh2ddtijonOJW+Psw6bcsx8MnjGRKzLD
G0ctdk6NFfaKtCvqc83PSaTBDgiko0YHTFXVhZk54ucMKgVduoUvuhdvGuFo
8zWtJHlZcmVvwtV4c+Cm8/NzcUosSZBg1MawRttaWhtNVOuLNzn4NoKI/LiX
V6HdBqRdV3HUM0S6Q62mbFJgTEAWQUgiTppPKtNIhKhpzS/GBkNMd4qWoZFX
hxvGLpDXXg3LrvlffFEsb1nLi5S3mCCYcRuTJtlQpwtcgnAYodXnXvZwQOtM
Q9sIJTLFzqKzjLQ8tFxnWsvA9ubMePx70++9jrRt5LHWCaiFDbBVNkc0sPNn
d9n1BAZ+tkadGw6xlgUvTiJhiohQ793Egw4IbYOzzRwtvHnD00NSKgmMzmIT
vWYYgq1XcFBolMUeAeyqTXDAnW6xNRRgoLqwITt0lT125JIG+V70M86Y+iry
tb50aAxOCCBiG0GjppLOvYKHoLbrUwQMCUVsrZfs34o7DZCIU1edZ3XiyVjA
6W5Tgx7NsHgjaS08tAwtgc6zYsrVCFyqXtaQCrArYjTTTscMzQWja8dB6RA2
Z3LqhU951mnF9YvbFu4jAiFyVhdPBq2rIFYcBLQpYJLceGMWx9oWG/LLOTdF
gKQwmy0IlQVLLHz1kLB+GEY/QR679KKfP61RuKe5rtguUAGMP2QxqPTrFuws
V4IhI5pLzR1O6uf4T6tVeYVEHwaKNki+DAl7YmrX21IO3yp6dZR7YiAsBGXS
zmL1pcaEvVaY7QjGCgDqkEkriJmmcyBfpaMRF8M0kjSa5RAi0wZMPqWel18M
I1l5xXapnPIbthIfkSX6kwp5yPdfbmJJNgX0j9N1gLSg25hA2F7yBdDBFCib
27BQBvCoDEbOHFrZOkNqxpu0QVyoY+wIJA0NXWsqt6nowSk5ibqVl2PaiuwJ
MoKxCwpGjAQZxCV8Yuy+Aynt9ZdR0EAMkkoJ6RNQAvoayoqZ7P2igxI/Kr3R
9SXjPLK/LqQHtw2DSMceN3CVImqyKIgHRSGxJxsscvWrvrGCFbq7xq8He1ba
SymQIQ2QitXa1lQ29UVBtiREQ9TAVYLJnXyvjxzz1jDu67eNdV4QG3rxYlB3
VYGijUhKGuFDL7wOaykNJshttwyVgAXmZ8uPVqaS37TlYLPuFJUo5BKdwdvW
kykinuy+SBzk4M9YJek/Gfk1ut25bLQLh6uL7VoI7s6fx/lzXvQyZOeDn4Ax
v477ezojCsYtBQ3CdaLNnekdGTmz186gojEYxSGhJOzv0bs8uLILx1hTS/FN
dMdm4zkL6TLHiMMWkXoRmpIytAYS3RyExY+dEVd3QaTN4WbzHlpb5bd7GOJ4
aPShTPVRgpsUzRdmuWG5DZT2otometdMo9hfDAt9OHPfCDreDMcYHUqB76R3
m+CoiyGhcvzdRPZdiHpjUFp8vK6XWR7xw4f48qBMpYoANe+QwRPLLc7aCvNG
MaS4aeVtkLoYIJi1NKmWTMzMSZ3Ii3lszWRQIBxJU76xl21USIDwqALdWwFX
PMOMvPbm5nWgsrpNwmnC6EU2xMc67gT0heSEg7TGIN5o4JE8FhE1PbWJiRUK
dBRee9A1pKZVyfwU0N9j8rDAID32Rh1qtTuU6hfVtmvfKWdESzTm7uysihVA
RBhl9HI2WZqURAwrNmQxnNKRyaVS5DMfXxVA2fWbEQnZqhmINe3xfcjX3VGI
ubmJrnEKGkQD1fUyjnR2ClN1ni++qz20VFU4r8IQWOoIyVkoHx8C+7rhZXhg
TImXssg+SOrnFXchSMTnpHIkLF8ca0PyVmSglkFnqIqOpMJVWpDg6lBbpd6H
okg1s9ZAdQLsNiGIEjrMKggpaJMBpIroVEAxV2FiG5VCSn0pwIRMCB5N3l8+
4flP18xkxG7+uCh9v98CB1PRyFYU5DRZxpAvc7Zp1uwGGxCyOTq6BMjyu4Zb
X1f5UoTLpxMsBl69lU/aSjhgb1Zz9SNCrbW1gUg/Dtkhip8X0dPJwNn+mG+j
rzeXFMirPJKjlzaZucBOrIXXGfO3xsDjH7HM+v7/nvdL55x8mlHOWkCDVatr
KBiATEsJN4LOzdycV/Fuo4y1V7GO1c6GiezAUgymhkV9J+xp1AtzjWzM4ftT
ywtCGkQhZjvPmP47ndd8dGLdrD0UMMWjJ+GcXi4PaAxEAAYKrjLqP00bajxg
eSyE2xoPfRYrxkKE8yoo7IlxRescxYZXCUSOaqBK3tldSawxWy2iYdxRp/h+
oBEvq9zvRGcFDNUA+8zMwfj6XO1Sck8uwQelEk3Q7UD5sTYFH18fv/B0x+mQ
gAD908nq2/K76eg2C53x96NpnMPj6fT3wLYP2wTbTG4xL/I/K8fXPiDJZTFn
gqIA+evw+OWmsyA/3BD4MgMYv/UAQURABgYbxWXV3M66VS/VN7io0WYRCwhw
Iv1ZTVUlbXLi/VVIyvGTrYQABYhBWMQdSHSNNtfh4VGCGHcmXeQZ5c+a+24D
EwVv0oHCRBAAfk4FA6b4U8iJO5EQHIEAAAHW6Jl0Hfneyg8x09qIJMDrAkuR
qhScUAflbD5o/N5HJ1nk1UY2Md2Rchnv+H8PVdgsK7YY/jSovLwyih9fo2DG
tUqPKAvVtpulWgieQIC0xCq9uyxJ48DHBoDCpqiKCvNdUa2qaw9t+ty2dt4t
+jnnPsQAAeva614Oqf0VgQAB0+p1wNc54oUzXxAelKZCS9quIEILgk0gQirS
EKLZS3v9K4+ggACCrKIZrFSID1yANh2kk/rp2mRSlyXQXNgbEhJwy0ODoCbw
86IDIvsmLB1zlIPH6i3H/2/MuvmS3z8dLrbQH4MbGMBsDkJMNsPqX/R5LWt+
fnrx4GkZOAETsGhB9dSL482fH8O05w9xAgCAr3HVhnyyYFHRg4BVze/xdyRT
hQkAl6ozQA==

--1171294248-1467159452-1199037483=:20553--
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Sun, 2007-12-30 at 10:58 -0700, dean gaudet wrote:
> On Sat, 29 Dec 2007, Dan Williams wrote:
>
> > On Dec 29, 2007 1:58 PM, dean gaudet wrote:
> > > On Sat, 29 Dec 2007, Dan Williams wrote:
> > >
> > > > On Dec 29, 2007 9:48 AM, dean gaudet wrote:
> > > > > hmm bummer, i'm doing another test (rsync 3.5M inodes from another box) on
> > > > > the same 64k chunk array and had raised the stripe_cache_size to 1024...
> > > > > and got a hang. this time i grabbed stripe_cache_active before bumping
> > > > > the size again -- it was only 905 active. as i recall the bug we were
> > > > > debugging a year+ ago the active was at the size when it would hang. so
> > > > > this is probably something new.
> > > >
> > > > I believe I am seeing the same issue and am trying to track down
> > > > whether XFS is doing something unexpected, i.e. I have not been able
> > > > to reproduce the problem with EXT3. MD tries to increase throughput
> > > > by letting some stripe work build up in batches. It looks like every
> > > > time your system has hung it has been in the 'inactive_blocked' state
> > > > i.e. > 3/4 of stripes active. This state should automatically
> > > > clear...
> > >
> > > cool, glad you can reproduce it :)
> > >
> > > i have a bit more data... i'm seeing the same problem on debian's
> > > 2.6.22-3-amd64 kernel, so it's not new in 2.6.24.
> > >
> >
> > This is just brainstorming at this point, but it looks like xfs can
> > submit more requests in the bi_end_io path such that it can lock
> > itself out of the RAID array. The sequence that concerns me is:
> >
> > return_io->xfs_buf_end_io->xfs_buf_io_end->xfs_buf_iodone_wo rk->xfs_buf_iorequest->make_request->
> >
> > I need verify whether this path is actually triggering, but if we are
> > in an inactive_blocked condition this new request will be put on a
> > wait queue and we'll never get to the release_stripe() call after
> > return_io(). It would be interesting to see if this is new XFS
> > behavior in recent kernels.
>
>
> i have evidence pointing to d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6 .git;a=commit;h=d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
>
> which was Neil's change in 2.6.22 for deferring generic_make_request
> until there's enough stack space for it.
>
> with my git tree sync'd to that commit my test cases fail in under 20
> minutes uptime (i rebooted and tested 3x). sync'd to the commit previous
> to it i've got 8h of run-time now without the problem.
>
> this isn't definitive of course since it does seem to be timing
> dependent, but since all failures have occured much earlier than that
> for me so far i think this indicates this change is either the cause of
> the problem or exacerbates an existing raid5 problem.
>
> given that this problem looks like a very rare problem i saw with 2.6.18
> (raid5+xfs there too) i'm thinking Neil's commit may just exacerbate an
> existing problem... not that i have evidence either way.
>
> i've attached a new kernel log with a hang at d89d87965d... and the
> reduced config file i was using for the bisect. hopefully the hang
> looks the same as what we were seeing at 2.6.24-rc6. let me know.
>

Dean could you try the below patch to see if it fixes your failure
scenario? It passes my test case.

Thanks,
Dan

------->
md: add generic_make_request_immed to prevent raid5 hang

From: Dan Williams

Commit d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1 reduced stack utilization
by preventing recursive calls to generic_make_request. However the
following conditions can cause raid5 to hang until 'stripe_cache_size' is
increased:

1/ stripe_cache_active is N stripes away from the 'inactive_blocked' limit
(3/4 * stripe_cache_size)
2/ a bio is submitted that requires M stripes to be processed where M > N
3/ stripes 1 through N are up-to-date and ready for immediate processing,
i.e. no trip through raid5d required

This results in the calling thread hanging while waiting for resources to
process stripes N through M. This means we never return from make_request.
All other raid5 users pile up in get_active_stripe. Increasing
stripe_cache_size temporarily resolves the blockage by allowing the blocked
make_request to return to generic_make_request.

Another way to solve this is to move all i/o submission to raid5d context.

Thanks to Dean Gaudet for bisecting this down to d89d8796.

Signed-off-by: Dan Williams
---

block/ll_rw_blk.c | 16 +++++++++++++---
drivers/md/raid5.c | 4 ++--
include/linux/blkdev.h | 1 +
3 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/block/ll_rw_blk.c b/block/ll_rw_blk.c
index 8b91994..bff40c2 100644
--- a/block/ll_rw_blk.c
+++ b/block/ll_rw_blk.c
@@ -3287,16 +3287,26 @@ end_io:
}

/*
- * We only want one ->make_request_fn to be active at a time,
- * else stack usage with stacked devices could be a problem.
+ * In the general case we only want one ->make_request_fn to be active
+ * at a time, else stack usage with stacked devices could be a problem.
* So use current->bio_{list,tail} to keep a list of requests
* submited by a make_request_fn function.
* current->bio_tail is also used as a flag to say if
* generic_make_request is currently active in this task or not.
* If it is NULL, then no make_request is active. If it is non-NULL,
* then a make_request is active, and new requests should be added
- * at the tail
+ * at the tail.
+ * However, some stacking drivers, like md-raid5, need to submit
+ * the bio without delay when it may not have the resources to
+ * complete its q->make_request_fn. generic_make_request_immed is
+ * provided for this explicit purpose.
*/
+void generic_make_request_immed(struct bio *bio)
+{
+ __generic_make_request(bio);
+}
+EXPORT_SYMBOL(generic_make_request_immed);
+
void generic_make_request(struct bio *bio)
{
if (current->bio_tail) {
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index c857b5a..ffa2be4 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -450,7 +450,7 @@ static void ops_run_io(struct stripe_head *sh)
test_bit(R5_ReWrite, &sh->dev[i].flags))
atomic_add(STRIPE_SECTORS,
&rdev->corrected_errors);
- generic_make_request(bi);
+ generic_make_request_immed(bi);
} else {
if (rw == WRITE)
set_bit(STRIPE_DEGRADED, &sh->state);
@@ -3124,7 +3124,7 @@ static void handle_stripe6(struct stripe_head *sh, struct page *tmp_page)
if (rw == WRITE &&
test_bit(R5_ReWrite, &sh->dev[i].flags))
atomic_add(STRIPE_SECTORS, &rdev->corrected_errors);
- generic_make_request(bi);
+ generic_make_request_immed(bi);
} else {
if (rw == WRITE)
set_bit(STRIPE_DEGRADED, &sh->state);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index d18ee67..774a3a0 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -642,6 +642,7 @@ extern int blk_register_queue(struct gendisk *disk);
extern void blk_unregister_queue(struct gendisk *disk);
extern void register_disk(struct gendisk *dev);
extern void generic_make_request(struct bio *bio);
+extern void generic_make_request_immed(struct bio *bio);
extern void blk_put_request(struct request *);
extern void __blk_put_request(struct request_queue *, struct request *);
extern void blk_end_sync_rq(struct request *rq, int error);


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Wednesday January 9, dan.j.williams@intel.com wrote:
> On Sun, 2007-12-30 at 10:58 -0700, dean gaudet wrote:
> > i have evidence pointing to d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> >
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6 .git;a=commit;h=d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> >
> > which was Neil's change in 2.6.22 for deferring generic_make_request
> > until there's enough stack space for it.
> >
>
> Commit d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1 reduced stack utilization
> by preventing recursive calls to generic_make_request. However the
> following conditions can cause raid5 to hang until 'stripe_cache_size' is
> increased:
>

Thanks for pursuing this guys. That explanation certainly sounds very
credible.

The generic_make_request_immed is a good way to confirm that we have
found the bug, but I don't like it as a long term solution, as it
just reintroduced the problem that we were trying to solve with the
problematic commit.

As you say, we could arrange that all request submission happens in
raid5d and I think this is the right way to proceed. However we can
still take some of the work into the thread that is submitting the
IO by calling "raid5d()" at the end of make_request, like this.

Can you test it please? Does it seem reasonable?

Thanks,
NeilBrown


Signed-off-by: Neil Brown

### Diffstat output
./drivers/md/md.c | 2 +-
./drivers/md/raid5.c | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)

diff .prev/drivers/md/md.c ./drivers/md/md.c
--- .prev/drivers/md/md.c 2008-01-07 13:32:10.000000000 +1100
+++ ./drivers/md/md.c 2008-01-10 11:08:02.000000000 +1100
@@ -5774,7 +5774,7 @@ void md_check_recovery(mddev_t *mddev)
if (mddev->ro)
return;

- if (signal_pending(current)) {
+ if (current == mddev->thread->tsk && signal_pending(current)) {
if (mddev->pers->sync_request) {
printk(KERN_INFO "md: %s in immediate safe mode\n",
mdname(mddev));

diff .prev/drivers/md/raid5.c ./drivers/md/raid5.c
--- .prev/drivers/md/raid5.c 2008-01-07 13:32:10.000000000 +1100
+++ ./drivers/md/raid5.c 2008-01-10 11:06:54.000000000 +1100
@@ -3432,6 +3432,7 @@ static int chunk_aligned_read(struct req
}
}

+static void raid5d (mddev_t *mddev);

static int make_request(struct request_queue *q, struct bio * bi)
{
@@ -3547,7 +3548,7 @@ static int make_request(struct request_q
goto retry;
}
finish_wait(&conf->wait_for_overlap, &w);
- handle_stripe(sh, NULL);
+ set_bit(STRIPE_HANDLE, &sh->state);
release_stripe(sh);
} else {
/* cannot get stripe for read-ahead, just give-up */
@@ -3569,6 +3570,7 @@ static int make_request(struct request_q
test_bit(BIO_UPTODATE, &bi->bi_flags)
? 0 : -EIO);
}
+ raid5d(mddev);
return 0;
}

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Jan 9, 2008 5:09 PM, Neil Brown wrote:
> On Wednesday January 9, dan.j.williams@intel.com wrote:
> > On Sun, 2007-12-30 at 10:58 -0700, dean gaudet wrote:
> > > i have evidence pointing to d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> > >
> > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6 .git;a=commit;h=d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> > >
> > > which was Neil's change in 2.6.22 for deferring generic_make_request
> > > until there's enough stack space for it.
> > >
> >
> > Commit d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1 reduced stack utilization
> > by preventing recursive calls to generic_make_request. However the
> > following conditions can cause raid5 to hang until 'stripe_cache_size' is
> > increased:
> >
>
> Thanks for pursuing this guys. That explanation certainly sounds very
> credible.
>
> The generic_make_request_immed is a good way to confirm that we have
> found the bug, but I don't like it as a long term solution, as it
> just reintroduced the problem that we were trying to solve with the
> problematic commit.
>
> As you say, we could arrange that all request submission happens in
> raid5d and I think this is the right way to proceed. However we can
> still take some of the work into the thread that is submitting the
> IO by calling "raid5d()" at the end of make_request, like this.
>
> Can you test it please?

This passes my failure case.

However, my test is different from Dean's in that I am using tiobench
and the latest rev of my 'get_priority_stripe' patch. I believe the
failure mechanism is the same, but it would be good to get
confirmation from Dean. get_priority_stripe has the effect of
increasing the frequency of
make_request->handle_stripe->generic_make_request sequences.

> Does it seem reasonable?

What do you think about limiting the number of stripes the submitting
thread handles to be equal to what it submitted? If I'm a stripe that
only submits 1 stripe worth of work should I get stuck handling the
rest of the cache?

Regards,
Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Wednesday January 9, dan.j.williams@intel.com wrote:
> On Jan 9, 2008 5:09 PM, Neil Brown wrote:
> > On Wednesday January 9, dan.j.williams@intel.com wrote:
> >
> > Can you test it please?
>
> This passes my failure case.

Thanks!

>
> > Does it seem reasonable?
>
> What do you think about limiting the number of stripes the submitting
> thread handles to be equal to what it submitted? If I'm a stripe that
> only submits 1 stripe worth of work should I get stuck handling the
> rest of the cache?

Dunno....
Someone has to do the work, and leaving it all to raid5d means that it
all gets done on one CPU.
I expect that most of the time the queue of ready stripes is empty so
make_request will mostly only handle it's own stripes anyway.
The times that it handles other thread's stripes will probably balance
out with the times that other threads handle this threads stripes.

So I'm incline to leave it as "do as much work as is available to be
done" as that is simplest. But I can probably be talked out of it
with a convincing argument....

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Wed, 2008-01-09 at 20:57 -0700, Neil Brown wrote:
> So I'm incline to leave it as "do as much work as is available to be
> done" as that is simplest. But I can probably be talked out of it
> with a convincing argument....

Well, in an age of CFS and CFQ it smacks of 'unfairness'. But does that
trump KISS...? Probably not.

--
Dan

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Thu, 10 Jan 2008, Neil Brown wrote:

> On Wednesday January 9, dan.j.williams@intel.com wrote:
> > On Sun, 2007-12-30 at 10:58 -0700, dean gaudet wrote:
> > > i have evidence pointing to d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> > >
> > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6 .git;a=commit;h=d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> > >
> > > which was Neil's change in 2.6.22 for deferring generic_make_request
> > > until there's enough stack space for it.
> > >
> >
> > Commit d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1 reduced stack utilization
> > by preventing recursive calls to generic_make_request. However the
> > following conditions can cause raid5 to hang until 'stripe_cache_size' is
> > increased:
> >
>
> Thanks for pursuing this guys. That explanation certainly sounds very
> credible.
>
> The generic_make_request_immed is a good way to confirm that we have
> found the bug, but I don't like it as a long term solution, as it
> just reintroduced the problem that we were trying to solve with the
> problematic commit.
>
> As you say, we could arrange that all request submission happens in
> raid5d and I think this is the right way to proceed. However we can
> still take some of the work into the thread that is submitting the
> IO by calling "raid5d()" at the end of make_request, like this.
>
> Can you test it please? Does it seem reasonable?


i've got this running now (against 2.6.24-rc6)... it has passed ~25
minutes of testing so far, which is a good sign. i'll report back
tomorrow and hopefully we'll have survived 8h+ of testing.

thanks!

w.r.t. dan's cfq comments -- i really don't know the details, but does
this mean cfq will misattribute the IO to the wrong user/process? or is
it just a concern that CPU time will be spent on someone's IO? the latter
is fine to me... the former seems sucky because with today's multicore
systems CPU time seems cheap compared to IO.

-dean
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Thu, 10 Jan 2008, Neil Brown wrote:

> On Wednesday January 9, dan.j.williams@intel.com wrote:
> > On Sun, 2007-12-30 at 10:58 -0700, dean gaudet wrote:
> > > i have evidence pointing to d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> > >
> > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6 .git;a=commit;h=d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1
> > >
> > > which was Neil's change in 2.6.22 for deferring generic_make_request
> > > until there's enough stack space for it.
> > >
> >
> > Commit d89d87965dcbe6fe4f96a2a7e8421b3a75f634d1 reduced stack utilization
> > by preventing recursive calls to generic_make_request. However the
> > following conditions can cause raid5 to hang until 'stripe_cache_size' is
> > increased:
> >
>
> Thanks for pursuing this guys. That explanation certainly sounds very
> credible.
>
> The generic_make_request_immed is a good way to confirm that we have
> found the bug, but I don't like it as a long term solution, as it
> just reintroduced the problem that we were trying to solve with the
> problematic commit.
>
> As you say, we could arrange that all request submission happens in
> raid5d and I think this is the right way to proceed. However we can
> still take some of the work into the thread that is submitting the
> IO by calling "raid5d()" at the end of make_request, like this.
>
> Can you test it please? Does it seem reasonable?
>
> Thanks,
> NeilBrown
>
>
> Signed-off-by: Neil Brown

it has passed 11h of the untar/diff/rm linux.tar.gz workload... that's
pretty good evidence it works for me. thanks!

Tested-by: dean gaudet

>
> ### Diffstat output
> ./drivers/md/md.c | 2 +-
> ./drivers/md/raid5.c | 4 +++-
> 2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff .prev/drivers/md/md.c ./drivers/md/md.c
> --- .prev/drivers/md/md.c 2008-01-07 13:32:10.000000000 +1100
> +++ ./drivers/md/md.c 2008-01-10 11:08:02.000000000 +1100
> @@ -5774,7 +5774,7 @@ void md_check_recovery(mddev_t *mddev)
> if (mddev->ro)
> return;
>
> - if (signal_pending(current)) {
> + if (current == mddev->thread->tsk && signal_pending(current)) {
> if (mddev->pers->sync_request) {
> printk(KERN_INFO "md: %s in immediate safe mode\n",
> mdname(mddev));
>
> diff .prev/drivers/md/raid5.c ./drivers/md/raid5.c
> --- .prev/drivers/md/raid5.c 2008-01-07 13:32:10.000000000 +1100
> +++ ./drivers/md/raid5.c 2008-01-10 11:06:54.000000000 +1100
> @@ -3432,6 +3432,7 @@ static int chunk_aligned_read(struct req
> }
> }
>
> +static void raid5d (mddev_t *mddev);
>
> static int make_request(struct request_queue *q, struct bio * bi)
> {
> @@ -3547,7 +3548,7 @@ static int make_request(struct request_q
> goto retry;
> }
> finish_wait(&conf->wait_for_overlap, &w);
> - handle_stripe(sh, NULL);
> + set_bit(STRIPE_HANDLE, &sh->state);
> release_stripe(sh);
> } else {
> /* cannot get stripe for read-ahead, just give-up */
> @@ -3569,6 +3570,7 @@ static int make_request(struct request_q
> test_bit(BIO_UPTODATE, &bi->bi_flags)
> ? 0 : -EIO);
> }
> + raid5d(mddev);
> return 0;
> }
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Jan 10, 2008 12:13 AM, dean gaudet wrote:
> w.r.t. dan's cfq comments -- i really don't know the details, but does
> this mean cfq will misattribute the IO to the wrong user/process? or is
> it just a concern that CPU time will be spent on someone's IO? the latter
> is fine to me... the former seems sucky because with today's multicore
> systems CPU time seems cheap compared to IO.
>

I do not see this affecting the time slicing feature of cfq, because
as Neil says the work has to get done at some point. If I give up
some of my slice working on someone else's I/O chances are the favor
will be returned in kind since the code does not discriminate. The
io-priority capability of cfq currently does not work as advertised
with current MD since the priority is tied to the current thread and
the thread that actually submits the i/o on a stripe is
non-deterministic. So I do not see this change making the situation
any worse. In fact, it may make it a bit better since there is a
higher chance for the thread submitting i/o to MD to do its own i/o to
the backing disks.

Reviewed-by: Dan Williams
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Neil Brown wrote:
> On Wednesday January 9, dan.j.williams@intel.com wrote:
>
>> On Jan 9, 2008 5:09 PM, Neil Brown wrote:
>>
>>> On Wednesday January 9, dan.j.williams@intel.com wrote:
>>>
>>> Can you test it please?
>>>
>> This passes my failure case.
>>
>
> Thanks!
>
>
>>> Does it seem reasonable?
>>>
>> What do you think about limiting the number of stripes the submitting
>> thread handles to be equal to what it submitted? If I'm a stripe that
>> only submits 1 stripe worth of work should I get stuck handling the
>> rest of the cache?
>>
>
> Dunno....
> Someone has to do the work, and leaving it all to raid5d means that it
> all gets done on one CPU.
> I expect that most of the time the queue of ready stripes is empty so
> make_request will mostly only handle it's own stripes anyway.
> The times that it handles other thread's stripes will probably balance
> out with the times that other threads handle this threads stripes.
>
> So I'm incline to leave it as "do as much work as is available to be
> done" as that is simplest. But I can probably be talked out of it
> with a convincing argument....
>

How about "it will perform better (defined as faster) during conditions
of unusual i/o activity?" Is that a convincing argument to use your
solution as offered? How about "complexity and maintainability are a
zero-sum problem?"

--
Bill Davidsen
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Thursday January 10, dan.j.williams@gmail.com wrote:
> On Jan 10, 2008 12:13 AM, dean gaudet wrote:
> > w.r.t. dan's cfq comments -- i really don't know the details, but does
> > this mean cfq will misattribute the IO to the wrong user/process? or is
> > it just a concern that CPU time will be spent on someone's IO? the latter
> > is fine to me... the former seems sucky because with today's multicore
> > systems CPU time seems cheap compared to IO.
> >
>
> I do not see this affecting the time slicing feature of cfq, because
> as Neil says the work has to get done at some point. If I give up
> some of my slice working on someone else's I/O chances are the favor
> will be returned in kind since the code does not discriminate. The
> io-priority capability of cfq currently does not work as advertised
> with current MD since the priority is tied to the current thread and
> the thread that actually submits the i/o on a stripe is
> non-deterministic. So I do not see this change making the situation
> any worse. In fact, it may make it a bit better since there is a
> higher chance for the thread submitting i/o to MD to do its own i/o to
> the backing disks.
>
> Reviewed-by: Dan Williams

Thanks.
But I suspect you didn't test it with a bitmap :-)
I ran the mdadm test suite and it hit a problem - easy enough to fix.

I'll look out for any other possible related problem (due to raid5d
running in different processes) and then submit it.

Thanks,
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

On Fri, 11 Jan 2008, Neil Brown wrote:

> Thanks.
> But I suspect you didn't test it with a bitmap :-)
> I ran the mdadm test suite and it hit a problem - easy enough to fix.

damn -- i "lost" my bitmap 'cause it was external and i didn't have things
set up properly to pick it up after a reboot :)

if you send an updated patch i'll give it another spin...

-dean
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Sorry if this breaks threaded mail readers, I only just subscribed to
the list so don;t have the original post to reply to.

I believe I'm having the same problem.

Regarding XFS on a raid5 md array:

Kernels 2.6.22-14 (Ubuntu Gutsy generic and server builds) *and*
2.6.24-rc8 (pure build from virgin sources) compiled for amd64 arch.

Raid 5 configured across 4 x 500GB SATA disks (Nforce nv_sata driver,
Asus M2N-E mobo, Athlon X64, 4GB RAM

MD Chunk size is 1024k. This is allocated to an LVM2 PV, then sliced up.
Taking one sample logical volume of 150GB I ran

mkfs.xfs -d su=1024k,sw=3 -L vol_linux /dev/vg00/vol_linux

I then found that putting high write load on that filesystem cause a
hang. High load could be a little as a single rsync of a mirror of
Ubunty Gutsy (many 10's of GB) from my old server to here. Hang would
happen in a few hours typically.

I could generate relatively quick hangs by running xfs_fsr (defragger)
in parallel.

Trying the workaround up upping /sys/block/md1/md/stripe_cache_size to
4096 seems (fingers crossed) to have helped. Been running the rsync
again, plus xfs_fst + a few dd's of 11 GB to the same filesystem.

I did notice also that the write speed increased dramatically with a
bigger stripe_cache_size.

A more detailed analysis of the problem indicated that, after the hang:

I could log in;

One CPU core was stuck in 100% IO wait.
The other core was useable, with care. So I managed to get a SysRQ T and
one place the system appeared blocked was via this path:

[ 2039.466258] xfs_fsr D 0000000000000000 0 7324 7308
[ 2039.466260] ffff810119399858 0000000000000082 0000000000000000
0000000000000046
[ 2039.466263] ffff810110d6c680 ffff8101102ba998 ffff8101102ba770
ffffffff8054e5e0
[ 2039.466265] ffff8101102ba998 000000010014a1e6 ffffffffffffffff
ffff810110ddcb30
[ 2039.466268] Call Trace:
[ 2039.466277] [] :raid456:get_active_stripe+0x1cb/0x610
[ 2039.466282] [] default_wake_function+0x0/0x10
[ 2039.466289] [] :raid456:make_request+0x1f8/0x610
[ 2039.466293] [] autoremove_wake_function+0x0/0x30
[ 2039.466295] [] __up_read+0x21/0xb0
[ 2039.466300] [] generic_make_request+0x1d6/0x3d0
[ 2039.466303] [] vm_normal_page+0x3d/0xc0
[ 2039.466307] [] submit_bio+0x6f/0xf0
[ 2039.466311] [] dio_bio_submit+0x5c/0x90
[ 2039.466313] [] dio_send_cur_page+0x43/0xa0
[ 2039.466316] [] submit_page_section+0x4e/0x150
[ 2039.466319] [] __blockdev_direct_IO+0x742/0xb50
[ 2039.466342] [] :xfs:xfs_vm_direct_IO+0x182/0x190
[ 2039.466357] [] :xfs:xfs_get_blocks_direct+0x0/0x20
[ 2039.466370] [] :xfs:xfs_end_io_direct+0x0/0x80
[ 2039.466375] [] __wait_on_bit_lock+0x65/0x80
[ 2039.466380] [] generic_file_direct_IO+0xe3/0x190
[ 2039.466385] [] generic_file_direct_write+0x74/0x150
[ 2039.466402] [] :xfs:xfs_write+0x492/0x8f0
[ 2039.466421] [] :xfs:xfs_iunlock+0x2c/0xb0
[ 2039.466437] [] :xfs:xfs_read+0x186/0x240
[ 2039.466443] [] do_sync_write+0xd9/0x120
[ 2039.466448] [] autoremove_wake_function+0x0/0x30
[ 2039.466457] [] vfs_write+0xdd/0x190
[ 2039.466461] [] sys_write+0x53/0x90
[ 2039.466465] [] system_call+0x7e/0x83


However, I'm of the opinion that the system should not deadlock, even if
tunable parameters are unfavourable. I'm happy with the workaround
(indeed the system performs better).

However, it will take me a week's worth of testing before I'm willing to
commission this as my new fileserver.

So, if there is anything anyone would like me to try, I'm happy to
volunteer as a guinea pig :)

Yes, I can build and patch kernels. But I'm not hot at debugging kernels
so if kernel core dumps or whatever are needed, please point me at the
right document or hint as to which commands I need to read about.

Cheers

Tim
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Tim Southerwood (ts@dionic.net) wrote on 23 January 2008 13:37:
>Sorry if this breaks threaded mail readers, I only just subscribed to
>the list so don;t have the original post to reply to.
>
>I believe I'm having the same problem.
>
>Regarding XFS on a raid5 md array:
>
>Kernels 2.6.22-14 (Ubuntu Gutsy generic and server builds) *and*
>2.6.24-rc8 (pure build from virgin sources) compiled for amd64 arch.

This has been corrected already, install Neil's patches. It worked for
several people under high stress, including us.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Carlos Carvalho wrote:
> Tim Southerwood (ts@dionic.net) wrote on 23 January 2008 13:37:
> >Sorry if this breaks threaded mail readers, I only just subscribed to
> >the list so don;t have the original post to reply to.
> >
> >I believe I'm having the same problem.
> >
> >Regarding XFS on a raid5 md array:
> >
> >Kernels 2.6.22-14 (Ubuntu Gutsy generic and server builds) *and*
> >2.6.24-rc8 (pure build from virgin sources) compiled for amd64 arch.
>
> This has been corrected already, install Neil's patches. It worked for
> several people under high stress, including us.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi

I just coerced the patch into 2.6.23.14, reset
/sys/block/md1/md/stripe_cache_size to default (256) and rebooted.

I can confirm that after 2 hours of heavy bashing[1] the system has not
hung. Looks good - many thanks. But I will run with a stripe_cache_size
of 4096 in practise as it improves write speen on my configuration about
2.5 times.

Cheers

Tim



[1] Rsync > 50GB to raid pluf xfs_fsr + dd 11GB of /dev/zero to same
filesystem.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Subtitle: Patch to mainline yet?

Hi

I don't see evidence of Neil's patch in 2.6.24, so I applied it by hand
on my server.

Was that the correct thing to do, or did this issue get fixed in a
different way that I wouldn't have spotted? I had a look at the git logs
but it was not obvious - please pardon my ignorance, I'm not familiar
enough with the code.

Many thanks,

Tim

Tim Southerwood wrote:
> Carlos Carvalho wrote:
>> Tim Southerwood (ts@dionic.net) wrote on 23 January 2008 13:37:
>> >Sorry if this breaks threaded mail readers, I only just subscribed
>> to >the list so don;t have the original post to reply to.
>> >
>> >I believe I'm having the same problem.
>> >
>> >Regarding XFS on a raid5 md array:
>> >
>> >Kernels 2.6.22-14 (Ubuntu Gutsy generic and server builds) *and*
>> >2.6.24-rc8 (pure build from virgin sources) compiled for amd64 arch.
>>
>> This has been corrected already, install Neil's patches. It worked for
>> several people under high stress, including us.
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> Hi
>
> I just coerced the patch into 2.6.23.14, reset
> /sys/block/md1/md/stripe_cache_size to default (256) and rebooted.
>
> I can confirm that after 2 hours of heavy bashing[1] the system has not
> hung. Looks good - many thanks. But I will run with a stripe_cache_size
> of 4096 in practise as it improves write speen on my configuration about
> 2.5 times.
>
> Cheers
>
> Tim
>
>
>
> [1] Rsync > 50GB to raid pluf xfs_fsr + dd 11GB of /dev/zero to same
> filesystem.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Tim Southerwood (ts@dionic.net) wrote on 28 January 2008 17:29:
>Subtitle: Patch to mainline yet?
>
>Hi
>
>I don't see evidence of Neil's patch in 2.6.24, so I applied it by hand
>on my server.

I applied all 4 pending patches to .24. It's been better than .22 and
..23... Unfortunately the bitmap and rai1 patch don't go in .22.16.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Carlos Carvalho wrote:
> Tim Southerwood (ts@dionic.net) wrote on 28 January 2008 17:29:
> >Subtitle: Patch to mainline yet?
> >
> >Hi
> >
> >I don't see evidence of Neil's patch in 2.6.24, so I applied it by hand
> >on my server.
>
> I applied all 4 pending patches to .24. It's been better than .22 and
> .23... Unfortunately the bitmap and rai1 patch don't go in .22.16.

Neil, have these been sent up against 24-stable and 23-stable?

--
Bill Davidsen
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: 2.6.24-rc6 reproducible raid5 hang

Am Dienstag, 29. Januar 2008 23:58 schrieb Bill Davidsen:
> Carlos Carvalho wrote:
> > Tim Southerwood (ts@dionic.net) wrote on 28 January 2008 17:29:
> > >Subtitle: Patch to mainline yet?
> > >
> > >Hi
> > >
> > >I don't see evidence of Neil's patch in 2.6.24, so I applied it
> > > by hand on my server.
> >
> > I applied all 4 pending patches to .24. It's been better than .22
> > and .23... Unfortunately the bitmap and rai1 patch don't go in
> > .22.16.
>
> Neil, have these been sent up against 24-stable and 23-stable?

... and .22-stable ?

Also, is this a xfs-on-raid5 bug or would it also happen with
ext3-on-raid5 ?

regards
Burkhard

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html