Hi mailing list,

First, thanks for this great software!

I have a RAID5 setup with on 6x 2TB HDD:

/dev/md0:
Version : 1.2
Creation Time : Tue Oct 19 08:58:41 2010
Raid Level : raid5
Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Tue Apr 12 17:50:25 2011
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : ion:0 (local to host ion)
UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
Events : 3035979

Number Major Minor RaidDevice State
0 8 81 0 active sync /dev/sdf1
1 8 17 1 active sync /dev/sdb1
4 8 49 2 active sync /dev/sdd1
3 8 33 3 active sync /dev/sdc1
5 8 65 4 active sync /dev/sde1
6 8 97 5 active sync /dev/sdg1

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[0] sdg1[6] sde1[5] sdc1[3] sdd1[4] sdb1[1]
9751756800 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
bitmap: 1/15 pages [4KB], 65536KB chunk

unused devices: <none>


I'm approaching over 6.5TB of data, and with an array this large I'd
like to migrate to RAID6 for a bit more safety. I'm just checking if I
understand this correctly, this is how to do it:

* Add a HDD to the array as a hot spare:
mdadm --manage /dev/md0 --add /dev/sdh1

* Migrate the array to RAID6:
mdadm --grow /dev/md0 --raid-devices 7 --level 6

Cheers,
// Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

--Sig_/uK_39il30GlX2G6T.fvMYWv
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Tue, 12 Apr 2011 17:56:05 +0100
Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wrote:

> I'm approaching over 6.5TB of data, and with an array this large I'd
> like to migrate to RAID6 for a bit more safety.

That's a great decision (and I suppose you made a typo in the subject).
RAID5 is downright dangerous at that disk count, and with disks of that siz=
e.

> I'm just checking if I
> understand this correctly, this is how to do it:
>
> * Add a HDD to the array as a hot spare:
> mdadm --manage /dev/md0 --add /dev/sdh1
>
> * Migrate the array to RAID6:
> mdadm --grow /dev/md0 --raid-devices 7 --level 6

Looks correct to me...

The first command can be just "mdadm --add /dev/md0 /dev/sdh1".

If you'd rather avoid a reshape at this point, you can add
"--layout=3Dpreserve" to the second line. That way you will have just a reb=
uild
of the new drive, instead of a full reshape.

You will also need to "--grow --bitmap=3Dnone" first (you can re-add the bi=
tmap
later).

--
With respect,
Roman

--Sig_/uK_39il30GlX2G6T.fvMYWv
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2kiFkACgkQTLKSvz+PZwjLGgCfXxmTsFm+jGZ1D/5XmysM q4rN
ifkAmgK8Iue40ikcLz+UfFX4ve+Qy9p0
=E1v7
-----END PGP SIGNATURE-----

--Sig_/uK_39il30GlX2G6T.fvMYWv--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 12 April 2011 18:14, Roman Mamedov <rm [at] romanrm.ru> wrote:
> On Tue, 12 Apr 2011 17:56:05 +0100
> Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wrote:
>
>> I'm approaching over 6.5TB of data, and with an array this large I'd
>> like to migrate to RAID6 for a bit more safety.
>
> That's a great decision (and I suppose you made a typo in the subject=
).
> RAID5 is downright dangerous at that disk count, and with disks of th=
at size.
>
>> I'm just checking if I
>> understand this correctly, this is how to do it:
>>
>> * Add a HDD to the array as a hot spare:
>> mdadm --manage /dev/md0 --add /dev/sdh1
>>
>> * Migrate the array to RAID6:
>> mdadm --grow /dev/md0 --raid-devices 7 --level 6
>
> Looks correct to me...
>
> The first command can be just "mdadm --add /dev/md0 /dev/sdh1".
>
> If you'd rather avoid a reshape at this point, you can add
> "--layout=3Dpreserve" to the second line. That way you will have just=
a rebuild
> of the new drive, instead of a full reshape.
>
> You will also need to "--grow --bitmap=3Dnone" first (you can re-add =
the bitmap
> later).
>
> --
> With respect,
> Roman
>

Hi,

Yep I mean RAID6, stupid subject line. If I use --layout=3Dpreserve ,
what impact will that have? Will the array have redundancy during the
rebuild of the new drive?
If I preserve the layout, what is the final result of the array
compared to not preserving it?

Cheers,
// Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

--Sig_/ssZLpGf_7=NkbA6H0UmdzAJ
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Tue, 12 Apr 2011 18:21:13 +0100
Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wrote:

> If I use --layout=3Dpreserve , what impact will that have?
> If I preserve the layout, what is the final result of the array
> compared to not preserving it?

Neil wrote about this on his blog:
"It is a very similar process that can now be used to convert a RAID5 to a
RAID6. We first change the RAID5 to RAID6 with a non-standard layout that h=
as
the parity blocks distributed as normal, but the Q blocks all on the last
device (a new device). So this is RAID6 using the RAID6 driver, but with a
non-RAID6 layout. So we "simply" change the layout and the job is done."
http://neil.brown.name/blog/20090817000931

Admittedly it is not completely clear to me what are the long-term downside=
s of
this layout. As I understand it does fully provide the RAID6-level redundan=
cy.
Perhaps just the performance will suffer a bit? Maybe someone can explain t=
his
more.

If anything, I think it is safe to use this layout for a while, e.g. in case
you don't want to rebuild 'right now'. You can always change the layout to =
the
traditional one later, by issuing "--grow --layout=3Dnormalise". Or perhaps=
if
you plan to add another disk soon, you can normalise it on that occasion, a=
nd
still gain the benefit of only one full reshape.

> Will the array have redundancy during the rebuild of the new drive?

If you choose --layout=3Dpreserve, your array immediately becomes a RAID6 w=
ith
one rebuilding drive. So this is the kind of redundancy you will have during
that rebuild - tolerance of up to one more (among the "old" drives) failure,
in other words, identical to what you currently have with RAID5.

--
With respect,
Roman

--Sig_/ssZLpGf_7=NkbA6H0UmdzAJ
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2kmG4ACgkQTLKSvz+PZwh+1ACfZ5AlZg/ZQpJ48nC3llGq mPLM
J+4Anj2ZdAkR4vXdKG59z9Z2wtm4aRPw
=X1gx
-----END PGP SIGNATURE-----

--Sig_/ssZLpGf_7=NkbA6H0UmdzAJ--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

--Sig_/bOzDzmoY7xHxe0zBmzkWxEM
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, 13 Apr 2011 00:22:38 +0600 Roman Mamedov <rm [at] romanrm.ru> wrote:

> On Tue, 12 Apr 2011 18:21:13 +0100
> Mathias Bur=E9n <mathias.buren [at] gmail.com> wrote:
>
> > If I use --layout=3Dpreserve , what impact will that have?
> > If I preserve the layout, what is the final result of the array
> > compared to not preserving it?
>
> Neil wrote about this on his blog:
> "It is a very similar process that can now be used to convert a RAID5 to a
> RAID6. We first change the RAID5 to RAID6 with a non-standard layout that=
has
> the parity blocks distributed as normal, but the Q blocks all on the last
> device (a new device). So this is RAID6 using the RAID6 driver, but with a
> non-RAID6 layout. So we "simply" change the layout and the job is done."
> http://neil.brown.name/blog/20090817000931
>
> Admittedly it is not completely clear to me what are the long-term downsi=
des of
> this layout. As I understand it does fully provide the RAID6-level redund=
ancy.
> Perhaps just the performance will suffer a bit? Maybe someone can explain=
this
> more.

If you specify --layout=3Dpreserve, then all the 'Q' blocks will be on one =
disk.
As every write needs to update a Q block, every write will write to that di=
sk.

With our current RAID6 implementation that probably isn't a big cost - for
any write, we need to either read from or write to each disk anyway.

Anyway: the only possible problem would be a performance problem, and I
really don't know what performance impact there is - if any.

>
> If anything, I think it is safe to use this layout for a while, e.g. in c=
ase
> you don't want to rebuild 'right now'. You can always change the layout t=
o the
> traditional one later, by issuing "--grow --layout=3Dnormalise". Or perha=
ps if
> you plan to add another disk soon, you can normalise it on that occasion,=
and
> still gain the benefit of only one full reshape.

Note that doing a normalise by itself later will be much slower than not
doing a preserve now.
Doing the normalise later when growing the the device again would be just as
fast as no doing the preserve now.

NeilBrown


>
> > Will the array have redundancy during the rebuild of the new drive?
>
> If you choose --layout=3Dpreserve, your array immediately becomes a RAID6=
with
> one rebuilding drive. So this is the kind of redundancy you will have dur=
ing
> that rebuild - tolerance of up to one more (among the "old" drives) failu=
re,
> in other words, identical to what you currently have with RAID5.
>


--Sig_/bOzDzmoY7xHxe0zBmzkWxEM
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)

iD8DBQFNpMEKG5fc6gV+Wb0RAi6pAJwM3W7XrMeBg3PC7x4mIviX3obZdACg 0fXk
uHuwg43UikEvv2Cdb7IK5EU=
=Txai
-----END PGP SIGNATURE-----

--Sig_/bOzDzmoY7xHxe0zBmzkWxEM--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 12 April 2011 22:15, NeilBrown <neilb [at] suse.de> wrote:
> On Wed, 13 Apr 2011 00:22:38 +0600 Roman Mamedov <rm [at] romanrm.ru> wrot=
e:
>
>> On Tue, 12 Apr 2011 18:21:13 +0100
>> Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wrote:
>>
>> > If I use --layout=3Dpreserve , what impact will that have?
>> > If I preserve the layout, what is the final result of the array
>> > compared to not preserving it?
>>
>> Neil wrote about this on his blog:
>> "It is a very similar process that can now be used to convert a RAID=
5 to a
>> RAID6. We first change the RAID5 to RAID6 with a non-standard layout=
that has
>> the parity blocks distributed as normal, but the Q blocks all on the=
last
>> device (a new device). So this is RAID6 using the RAID6 driver, but =
with a
>> non-RAID6 layout. So we "simply" change the layout and the job is do=
ne."
>> http://neil.brown.name/blog/20090817000931
>>
>> Admittedly it is not completely clear to me what are the long-term d=
ownsides of
>> this layout. As I understand it does fully provide the RAID6-level r=
edundancy.
>> Perhaps just the performance will suffer a bit? Maybe someone can ex=
plain this
>> more.
>
> If you specify --layout=3Dpreserve, then all the 'Q' blocks will be o=
n one disk.
> As every write needs to update a Q block, every write will write to t=
hat disk.
>
> With our current RAID6 implementation that probably isn't a big cost =
- for
> any write, we need to either read from or write to each disk anyway.
>
> Anyway: =C2=A0the only possible problem would be a performance proble=
m, and I
> really don't know what performance impact there is - if any.
>
>>
>> If anything, I think it is safe to use this layout for a while, e.g.=
in case
>> you don't want to rebuild 'right now'. You can always change the lay=
out to the
>> traditional one later, by issuing "--grow --layout=3Dnormalise". Or =
perhaps if
>> you plan to add another disk soon, you can normalise it on that occa=
sion, and
>> still gain the benefit of only one full reshape.
>
> Note that doing a normalise by itself later will be much slower than =
not
> doing a preserve now.
> Doing the normalise later when growing the the device again would be =
just as
> fast as no doing the preserve now.
>
> NeilBrown
>
>
>>
>> > =C2=A0Will the array have redundancy during the rebuild of the new=
drive?
>>
>> If you choose --layout=3Dpreserve, your array immediately becomes a =
RAID6 with
>> one rebuilding drive. So this is the kind of redundancy you will hav=
e during
>> that rebuild - tolerance of up to one more (among the "old" drives) =
failure,
>> in other words, identical to what you currently have with RAID5.
>>
>
>

Right, so using --preserve seems like a sane and good option. Thanks
for the info, I'll let you know what happens, HDD should arrive the
next few days.

// Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

(Subject line amended by me :-)

On 12/04/2011 17:56, Mathias Bur=C3=A9n wrote:
[...]
> I'm approaching over 6.5TB of data, and with an array this large I'd
> like to migrate to RAID6 for a bit more safety. I'm just checking if =
I
> understand this correctly, this is how to do it:
>
> * Add a HDD to the array as a hot spare:
> mdadm --manage /dev/md0 --add /dev/sdh1
>
> * Migrate the array to RAID6:
> mdadm --grow /dev/md0 --raid-devices 7 --level 6

You will need a --backup-file to do this, on another device. Since you
are keeping the same number of data discs before and after the reshape,=

the backup file will be needed throughout the reshape, so the reshape
will take perhaps twice as long as a grow or shrink. If your backup-fil=
e
is on the same disc(s) as md0 is (e.g. on another partition or array
made up of other partitions on the same disc(s)), it will take way
longer (gazillions of seeks), so I'd recommend a separate drive or if
you have one a small SSD for the backup file.

Doing the above with --layout=3Dpreserve will save you doing the reshap=
e
so you won't need the backup file, but there will still be an initial
sync of the Q parity, and the layout will be RAID4-alike with all the Q=

parity on one drive so it's possible its performance will be RAID4-alik=
e
too i.e. small writes never faster than the parity drive. Having said
that, streamed writes can still potentially go as fast as your 5 data
discs, as per your RAID5. In practice, I'd be surprised if it was faste=
r
than about twice the speed of a single drive (the same as your current
RAID5), and as Neil Brown notes in his reply, RAID6 doesn't currently
have the read-modify-write optimisation for small writes so small write=

performance is liable to be even poorer than your RAID5 in either layou=
t.

You will never lose any redundancy in either of the above, but you won'=
t
gain RAID6 double redundancy until the reshape (or Q-drive sync with
--layout=3Dpreserve) has completed - just the same as if you were
replacing a dead drive in an existing RAID6.

Hope the above helps!

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 13 April 2011 12:44, John Robinson <john.robinson [at] anonymous.org.uk> =
wrote:
> (Subject line amended by me :-)
>
> On 12/04/2011 17:56, Mathias Bur=C3=A9n wrote:
> [...]
>>
>> I'm approaching over 6.5TB of data, and with an array this large I'd
>> like to migrate to RAID6 for a bit more safety. I'm just checking if=
I
>> understand this correctly, this is how to do it:
>>
>> * Add a HDD to the array as a hot spare:
>> mdadm --manage /dev/md0 --add /dev/sdh1
>>
>> * Migrate the array to RAID6:
>> mdadm --grow /dev/md0 --raid-devices 7 --level 6
>
> You will need a --backup-file to do this, on another device. Since yo=
u are
> keeping the same number of data discs before and after the reshape, t=
he
> backup file will be needed throughout the reshape, so the reshape wil=
l take
> perhaps twice as long as a grow or shrink. If your backup-file is on =
the
> same disc(s) as md0 is (e.g. on another partition or array made up of=
other
> partitions on the same disc(s)), it will take way longer (gazillions =
of
> seeks), so I'd recommend a separate drive or if you have one a small =
SSD for
> the backup file.
>
> Doing the above with --layout=3Dpreserve will save you doing the resh=
ape so
> you won't need the backup file, but there will still be an initial sy=
nc of
> the Q parity, and the layout will be RAID4-alike with all the Q parit=
y on
> one drive so it's possible its performance will be RAID4-alike too i.=
e.
> small writes never faster than the parity drive. Having said that, st=
reamed
> writes can still potentially go as fast as your 5 data discs, as per =
your
> RAID5. In practice, I'd be surprised if it was faster than about twic=
e the
> speed of a single drive (the same as your current RAID5), and as Neil=
Brown
> notes in his reply, RAID6 doesn't currently have the read-modify-writ=
e
> optimisation for small writes so small write performance is liable to=
be
> even poorer than your RAID5 in either layout.
>
> You will never lose any redundancy in either of the above, but you wo=
n't
> gain RAID6 double redundancy until the reshape (or Q-drive sync with
> --layout=3Dpreserve) has completed - just the same as if you were rep=
lacing a
> dead drive in an existing RAID6.
>
> Hope the above helps!
>
> Cheers,
>
> John.
>
>

Hi,

Thanks for the replies. Allright, here we go;

$ mdadm --grow /dev/md0 --bitmap=3Dnone
$ mdadm --manage /dev/md0 --add /dev/sde1
$ mdadm --grow /dev/md0 --verbose --layout=3Dpreserve --raid-devices =
7
--level 6 --backup-file=3D/root/md-raid5-to-raid6-backupfile.bin
mdadm: level of /dev/md0 changed to raid6

$ cat /proc/mdstat

Fri Apr
22 10:37:44 2011

Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sde1[7] sdg1[0] sdh1[6] sdf1[5] sdc1[3] sdd1[4] sdb1=
[1]
9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 18
[7/6] [UUUUUU_]
[>....................] reshape =3D 0.0% (224768/1950351360)
finish=3D8358.5min speed=3D3888K/sec

unused devices: <none>

And in dmesg:


--- level:6 rd:7 wd:6
disk 0, o:1, dev:sdg1
disk 1, o:1, dev:sdb1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdc1
disk 4, o:1, dev:sdf1
disk 5, o:1, dev:sdh1
RAID conf printout:
--- level:6 rd:7 wd:6
disk 0, o:1, dev:sdg1
disk 1, o:1, dev:sdb1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdc1
disk 4, o:1, dev:sdf1
disk 5, o:1, dev:sdh1
disk 6, o:1, dev:sde1
md: reshape of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than
200000 KB/sec) for reshape.
md: using 128k window, over a total of 1950351360 blocks.

IIRC there's a way to speed up the migration, by using a larger cache
value somewhere, no?

Thanks,
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 22 April 2011 10:39, Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wr=
ote:
> On 13 April 2011 12:44, John Robinson <john.robinson [at] anonymous.org.uk=
> wrote:
>> (Subject line amended by me :-)
>>
>> On 12/04/2011 17:56, Mathias Bur=C3=A9n wrote:
>> [...]
>>>
>>> I'm approaching over 6.5TB of data, and with an array this large I'=
d
>>> like to migrate to RAID6 for a bit more safety. I'm just checking i=
f I
>>> understand this correctly, this is how to do it:
>>>
>>> * Add a HDD to the array as a hot spare:
>>> mdadm --manage /dev/md0 --add /dev/sdh1
>>>
>>> * Migrate the array to RAID6:
>>> mdadm --grow /dev/md0 --raid-devices 7 --level 6
>>
>> You will need a --backup-file to do this, on another device. Since y=
ou are
>> keeping the same number of data discs before and after the reshape, =
the
>> backup file will be needed throughout the reshape, so the reshape wi=
ll take
>> perhaps twice as long as a grow or shrink. If your backup-file is on=
the
>> same disc(s) as md0 is (e.g. on another partition or array made up o=
f other
>> partitions on the same disc(s)), it will take way longer (gazillions=
of
>> seeks), so I'd recommend a separate drive or if you have one a small=
SSD for
>> the backup file.
>>
>> Doing the above with --layout=3Dpreserve will save you doing the res=
hape so
>> you won't need the backup file, but there will still be an initial s=
ync of
>> the Q parity, and the layout will be RAID4-alike with all the Q pari=
ty on
>> one drive so it's possible its performance will be RAID4-alike too i=
=2Ee.
>> small writes never faster than the parity drive. Having said that, s=
treamed
>> writes can still potentially go as fast as your 5 data discs, as per=
your
>> RAID5. In practice, I'd be surprised if it was faster than about twi=
ce the
>> speed of a single drive (the same as your current RAID5), and as Nei=
l Brown
>> notes in his reply, RAID6 doesn't currently have the read-modify-wri=
te
>> optimisation for small writes so small write performance is liable t=
o be
>> even poorer than your RAID5 in either layout.
>>
>> You will never lose any redundancy in either of the above, but you w=
on't
>> gain RAID6 double redundancy until the reshape (or Q-drive sync with
>> --layout=3Dpreserve) has completed - just the same as if you were re=
placing a
>> dead drive in an existing RAID6.
>>
>> Hope the above helps!
>>
>> Cheers,
>>
>> John.
>>
>>
>
> Hi,
>
> Thanks for the replies. Allright, here we go;
>
> =C2=A0$ mdadm --grow /dev/md0 --bitmap=3Dnone
> =C2=A0$ mdadm --manage /dev/md0 --add /dev/sde1
> =C2=A0$ mdadm --grow /dev/md0 --verbose --layout=3Dpreserve =C2=A0--r=
aid-devices 7
> --level 6 --backup-file=3D/root/md-raid5-to-raid6-backupfile.bin
> mdadm: level of /dev/md0 changed to raid6
>
> $ cat /proc/mdstat
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 F=
ri Apr
> 22 10:37:44 2011
>
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid6 sde1[7] sdg1[0] sdh1[6] sdf1[5] sdc1[3] sdd1[4] sd=
b1[1]
> =C2=A0 =C2=A0 =C2=A09751756800 blocks super 1.2 level 6, 64k chunk, a=
lgorithm 18
> [7/6] [UUUUUU_]
> =C2=A0 =C2=A0 =C2=A0[>....................] =C2=A0reshape =3D =C2=A00=
=2E0% (224768/1950351360)
> finish=3D8358.5min speed=3D3888K/sec
>
> unused devices: <none>
>
> And in dmesg:
>
>
> =C2=A0--- level:6 rd:7 wd:6
> =C2=A0disk 0, o:1, dev:sdg1
> =C2=A0disk 1, o:1, dev:sdb1
> =C2=A0disk 2, o:1, dev:sdd1
> =C2=A0disk 3, o:1, dev:sdc1
> =C2=A0disk 4, o:1, dev:sdf1
> =C2=A0disk 5, o:1, dev:sdh1
> RAID conf printout:
> =C2=A0--- level:6 rd:7 wd:6
> =C2=A0disk 0, o:1, dev:sdg1
> =C2=A0disk 1, o:1, dev:sdb1
> =C2=A0disk 2, o:1, dev:sdd1
> =C2=A0disk 3, o:1, dev:sdc1
> =C2=A0disk 4, o:1, dev:sdf1
> =C2=A0disk 5, o:1, dev:sdh1
> =C2=A0disk 6, o:1, dev:sde1
> md: reshape of RAID array md0
> md: minimum _guaranteed_ =C2=A0speed: 1000 KB/sec/disk.
> md: using maximum available idle IO bandwidth (but not more than
> 200000 KB/sec) for reshape.
> md: using 128k window, over a total of 1950351360 blocks.
>
> IIRC there's a way to speed up the migration, by using a larger cache
> value somewhere, no?
>
> Thanks,
> Mathias
>

Increasing stripe cache on the md device from 1027 to 32k or 16k
didn't make a difference, still around 3800KB/s reshape. Oh well,
we'll see if it's still alive in 5.5 days!

Cheers,
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 22 April 2011 11:05, Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wr=
ote:
> On 22 April 2011 10:39, Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> =
wrote:
>> On 13 April 2011 12:44, John Robinson <john.robinson [at] anonymous.org.u=
k> wrote:
>>> (Subject line amended by me :-)
>>>
>>> On 12/04/2011 17:56, Mathias Bur=C3=A9n wrote:
>>> [...]
>>>>
>>>> I'm approaching over 6.5TB of data, and with an array this large I=
'd
>>>> like to migrate to RAID6 for a bit more safety. I'm just checking =
if I
>>>> understand this correctly, this is how to do it:
>>>>
>>>> * Add a HDD to the array as a hot spare:
>>>> mdadm --manage /dev/md0 --add /dev/sdh1
>>>>
>>>> * Migrate the array to RAID6:
>>>> mdadm --grow /dev/md0 --raid-devices 7 --level 6
>>>
>>> You will need a --backup-file to do this, on another device. Since =
you are
>>> keeping the same number of data discs before and after the reshape,=
the
>>> backup file will be needed throughout the reshape, so the reshape w=
ill take
>>> perhaps twice as long as a grow or shrink. If your backup-file is o=
n the
>>> same disc(s) as md0 is (e.g. on another partition or array made up =
of other
>>> partitions on the same disc(s)), it will take way longer (gazillion=
s of
>>> seeks), so I'd recommend a separate drive or if you have one a smal=
l SSD for
>>> the backup file.
>>>
>>> Doing the above with --layout=3Dpreserve will save you doing the re=
shape so
>>> you won't need the backup file, but there will still be an initial =
sync of
>>> the Q parity, and the layout will be RAID4-alike with all the Q par=
ity on
>>> one drive so it's possible its performance will be RAID4-alike too =
i.e.
>>> small writes never faster than the parity drive. Having said that, =
streamed
>>> writes can still potentially go as fast as your 5 data discs, as pe=
r your
>>> RAID5. In practice, I'd be surprised if it was faster than about tw=
ice the
>>> speed of a single drive (the same as your current RAID5), and as Ne=
il Brown
>>> notes in his reply, RAID6 doesn't currently have the read-modify-wr=
ite
>>> optimisation for small writes so small write performance is liable =
to be
>>> even poorer than your RAID5 in either layout.
>>>
>>> You will never lose any redundancy in either of the above, but you =
won't
>>> gain RAID6 double redundancy until the reshape (or Q-drive sync wit=
h
>>> --layout=3Dpreserve) has completed - just the same as if you were r=
eplacing a
>>> dead drive in an existing RAID6.
>>>
>>> Hope the above helps!
>>>
>>> Cheers,
>>>
>>> John.
>>>
>>>
>>
>> Hi,
>>
>> Thanks for the replies. Allright, here we go;
>>
>> =C2=A0$ mdadm --grow /dev/md0 --bitmap=3Dnone
>> =C2=A0$ mdadm --manage /dev/md0 --add /dev/sde1
>> =C2=A0$ mdadm --grow /dev/md0 --verbose --layout=3Dpreserve =C2=A0--=
raid-devices 7
>> --level 6 --backup-file=3D/root/md-raid5-to-raid6-backupfile.bin
>> mdadm: level of /dev/md0 changed to raid6
>>
>> $ cat /proc/mdstat
>>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 F=
ri Apr
>> 22 10:37:44 2011
>>
>> Personalities : [raid6] [raid5] [raid4]
>> md0 : active raid6 sde1[7] sdg1[0] sdh1[6] sdf1[5] sdc1[3] sdd1[4] s=
db1[1]
>> =C2=A0 =C2=A0 =C2=A09751756800 blocks super 1.2 level 6, 64k chunk, =
algorithm 18
>> [7/6] [UUUUUU_]
>> =C2=A0 =C2=A0 =C2=A0[>....................] =C2=A0reshape =3D =C2=A0=
0.0% (224768/1950351360)
>> finish=3D8358.5min speed=3D3888K/sec
>>
>> unused devices: <none>
>>
>> And in dmesg:
>>
>>
>> =C2=A0--- level:6 rd:7 wd:6
>> =C2=A0disk 0, o:1, dev:sdg1
>> =C2=A0disk 1, o:1, dev:sdb1
>> =C2=A0disk 2, o:1, dev:sdd1
>> =C2=A0disk 3, o:1, dev:sdc1
>> =C2=A0disk 4, o:1, dev:sdf1
>> =C2=A0disk 5, o:1, dev:sdh1
>> RAID conf printout:
>> =C2=A0--- level:6 rd:7 wd:6
>> =C2=A0disk 0, o:1, dev:sdg1
>> =C2=A0disk 1, o:1, dev:sdb1
>> =C2=A0disk 2, o:1, dev:sdd1
>> =C2=A0disk 3, o:1, dev:sdc1
>> =C2=A0disk 4, o:1, dev:sdf1
>> =C2=A0disk 5, o:1, dev:sdh1
>> =C2=A0disk 6, o:1, dev:sde1
>> md: reshape of RAID array md0
>> md: minimum _guaranteed_ =C2=A0speed: 1000 KB/sec/disk.
>> md: using maximum available idle IO bandwidth (but not more than
>> 200000 KB/sec) for reshape.
>> md: using 128k window, over a total of 1950351360 blocks.
>>
>> IIRC there's a way to speed up the migration, by using a larger cach=
e
>> value somewhere, no?
>>
>> Thanks,
>> Mathias
>>
>
> Increasing stripe cache on the md device from 1027 to 32k or 16k
> didn't make a difference, still around 3800KB/s reshape. Oh well,
> we'll see if it's still alive in 5.5 days!
>
> Cheers,
>

On 22 April 2011 11:05, Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> wr=
ote:
> On 22 April 2011 10:39, Mathias Bur=C3=A9n <mathias.buren [at] gmail.com> =
wrote:
>> On 13 April 2011 12:44, John Robinson <john.robinson [at] anonymous.org.u=
k> wrote:
>>> (Subject line amended by me :-)
>>>
>>> On 12/04/2011 17:56, Mathias Bur=C3=A9n wrote:
>>> [...]
>>>>
>>>> I'm approaching over 6.5TB of data, and with an array this large I=
'd
>>>> like to migrate to RAID6 for a bit more safety. I'm just checking =
if I
>>>> understand this correctly, this is how to do it:
>>>>
>>>> * Add a HDD to the array as a hot spare:
>>>> mdadm --manage /dev/md0 --add /dev/sdh1
>>>>
>>>> * Migrate the array to RAID6:
>>>> mdadm --grow /dev/md0 --raid-devices 7 --level 6
>>>
>>> You will need a --backup-file to do this, on another device. Since =
you are
>>> keeping the same number of data discs before and after the reshape,=
the
>>> backup file will be needed throughout the reshape, so the reshape w=
ill take
>>> perhaps twice as long as a grow or shrink. If your backup-file is o=
n the
>>> same disc(s) as md0 is (e.g. on another partition or array made up =
of other
>>> partitions on the same disc(s)), it will take way longer (gazillion=
s of
>>> seeks), so I'd recommend a separate drive or if you have one a smal=
l SSD for
>>> the backup file.
>>>
>>> Doing the above with --layout=3Dpreserve will save you doing the re=
shape so
>>> you won't need the backup file, but there will still be an initial =
sync of
>>> the Q parity, and the layout will be RAID4-alike with all the Q par=
ity on
>>> one drive so it's possible its performance will be RAID4-alike too =
i.e.
>>> small writes never faster than the parity drive. Having said that, =
streamed
>>> writes can still potentially go as fast as your 5 data discs, as pe=
r your
>>> RAID5. In practice, I'd be surprised if it was faster than about tw=
ice the
>>> speed of a single drive (the same as your current RAID5), and as Ne=
il Brown
>>> notes in his reply, RAID6 doesn't currently have the read-modify-wr=
ite
>>> optimisation for small writes so small write performance is liable =
to be
>>> even poorer than your RAID5 in either layout.
>>>
>>> You will never lose any redundancy in either of the above, but you =
won't
>>> gain RAID6 double redundancy until the reshape (or Q-drive sync wit=
h
>>> --layout=3Dpreserve) has completed - just the same as if you were r=
eplacing a
>>> dead drive in an existing RAID6.
>>>
>>> Hope the above helps!
>>>
>>> Cheers,
>>>
>>> John.
>>>
>>>
>>
>> Hi,
>>
>> Thanks for the replies. Allright, here we go;
>>
>> $ mdadm --grow /dev/md0 --bitmap=3Dnone
>> $ mdadm --manage /dev/md0 --add /dev/sde1
>> $ mdadm --grow /dev/md0 --verbose --layout=3Dpreserve --raid-devic=
es 7
>> --level 6 --backup-file=3D/root/md-raid5-to-raid6-backupfile.bin
>> mdadm: level of /dev/md0 changed to raid6
>>
>> $ cat /proc/mdstat
>>
>> Fri Apr
>> 22 10:37:44 2011
>>
>> Personalities : [raid6] [raid5] [raid4]
>> md0 : active raid6 sde1[7] sdg1[0] sdh1[6] sdf1[5] sdc1[3] sdd1[4] s=
db1[1]
>> 9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 18
>> [7/6] [UUUUUU_]
>> [>....................] reshape =3D 0.0% (224768/1950351360)
>> finish=3D8358.5min speed=3D3888K/sec
>>
>> unused devices: <none>
>>
>> And in dmesg:
>>
>>
>> --- level:6 rd:7 wd:6
>> disk 0, o:1, dev:sdg1
>> disk 1, o:1, dev:sdb1
>> disk 2, o:1, dev:sdd1
>> disk 3, o:1, dev:sdc1
>> disk 4, o:1, dev:sdf1
>> disk 5, o:1, dev:sdh1
>> RAID conf printout:
>> --- level:6 rd:7 wd:6
>> disk 0, o:1, dev:sdg1
>> disk 1, o:1, dev:sdb1
>> disk 2, o:1, dev:sdd1
>> disk 3, o:1, dev:sdc1
>> disk 4, o:1, dev:sdf1
>> disk 5, o:1, dev:sdh1
>> disk 6, o:1, dev:sde1
>> md: reshape of RAID array md0
>> md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
>> md: using maximum available idle IO bandwidth (but not more than
>> 200000 KB/sec) for reshape.
>> md: using 128k window, over a total of 1950351360 blocks.
>>
>> IIRC there's a way to speed up the migration, by using a larger cach=
e
>> value somewhere, no?
>>
>> Thanks,
>> Mathias
>>
>
> Increasing stripe cache on the md device from 1027 to 32k or 16k
> didn't make a difference, still around 3800KB/s reshape. Oh well,
> we'll see if it's still alive in 5.5 days!
>
> Cheers,
>

It's alive!

md: md0: reshape done.
RAID conf printout:
--- level:6 rd:7 wd:7
disk 0, o:1, dev:sdg1
disk 1, o:1, dev:sdb1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdc1
disk 4, o:1, dev:sdf1
disk 5, o:1, dev:sdh1
disk 6, o:1, dev:sde1

$ sudo mdadm -D /dev/md0
Password:
/dev/md0:
Version : 1.2
Creation Time : Tue Oct 19 08:58:41 2010
Raid Level : raid6
Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent

Update Time : Fri Apr 29 23:44:50 2011
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : ion:0 (local to host ion)
UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
Events : 6158702

Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 17 1 active sync /dev/sdb1
4 8 49 2 active sync /dev/sdd1
3 8 33 3 active sync /dev/sdc1
5 8 81 4 active sync /dev/sdf1
6 8 113 5 active sync /dev/sdh1
7 8 65 6 active sync /dev/sde1

Yay :) thanks for rgeat software! Cheers,

/ Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo [at] vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html