embedding area is unusually small... (GRUB2 on software RAID1)

Hello,

I successfully set up an Intel Matrix Raid device with a RAID1 and a RAID0 volume, each having a couple of partitions, but then I could not install GRUB2 on the RAID1 volume, which I wanted to use to boot from and mount as root. It turned out that the "IMSM" metadata is not supported in GRUB2 (v1.97.1) just yet, so I had to turn away from my original plan.

To "imitate" the setup I originally wanded, I turned both of my drives into AHCI controlled devices in the BIOS (instead of RAID), and I partitioned them to obtain /dev/sda[12] and /dev/sdb[12].

Then I used /dev/sd[ab]1 to build a RAID1 set, and /dev/sd[ab]2 to create a RAID0 set using mdadm v 3.0.3:

> mdadm -C /dev/md0 -v -e 0 -l 1 -n 2 /dev/sda1 /dev/sdb1
> mdadm -C /dev/md1 -v -e 0 -l 0 -n 2 /dev/sda2 /dev/sdb2

I set the metadata type to 0.90 because I would like to boot from it and allow the kernel to auto-detect the RAID devices while it's booting, in order to can get away from using an intitrd (I am building my own distribution based on CLFS x86_64 multilib).

I used cfdisk to partition both of the /dev/md[01] devices, and I obtained /dev/md0p[123] and /dev/md1p[12]. The plan is to use /dev/md0p1 as a RAID1 root partition, and have the system boot from /dev/md0. I formatted /dev/md0p1 as

> mk2efs -t ext4 -L OS /dev/md0p1

To this point, things went smoothly. mdadm -D... and mdadm -E... did report back working devices as intended. Then mounted /dev/md0p1 on a directory called /root/os, and I did

> grub-install --root-directory=/root/os /dev/md0

or

> grub-install --root-directory=/root/os "md0"

and I got a warning and an error message: "Your embedding area is unusually small. core.img won't fit in it." and "Embedding is not possible, but this is required when the root device is on a RAID array or LVM volume."

What did I do wrong, and how do I fix it? Thanks ahead,
Tibor


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Lapohos Tibor wrote:

[...]
>> grub-install --root-directory=/root/os /dev/md0

> and I got a warning and an error message: "Your embedding area is
> unusually small. core.img won't fit in it." and "Embedding is not
> possible, but this is required when the root device is on a RAID array or
> LVM volume."

> What did I do wrong, and how do I fix it? Thanks ahead,

Grub2 needs a space of about 55 sectors free after the MBR. It ussually is
63 sectors. What's the output of:
fdisk -lu /dev/sda
the start of the first partition should be 63 sectors or bigger to leave
enough space for grub2 to be embedded.

However, the way you are making partitions is quite complex and the needed
space could be somewhere else.

Please, stop repeating your question everywhere.

--
Antonio Perez

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Thanks for your reply, Antonio.

--- On Wed, 1/6/10, Antonio Perez wrote:

> What's the output of:
>   =A0 fdisk -lu /dev/sda

I will look at it tonight. Thanks.

> However, the way you are making partitions is quite complex
> and the needed space could be somewhere else.

Does this mean that this partitioning is not the right way to proceed? =
should I create all the partitions first and build them into so many /d=
ev/md[01234] devices separately maybe?

> Please, stop repeating your question everywhere.

I am very sorry if I upset the community. I did not think that linux-ra=
id, grub-devel and help-grub lists would represent the same experts. In=
fact, based on what I see on the GRUB sites is, that grub-devel expert=
s are different from more like "legacy" supporting help-grub ones. Mayb=
e a bit of clarification would help keeping the developers/maintainers =
email intake a bit lower. So I must ask: which one is the most appropri=
ate for my question?

Thanks again,
Tibor

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Lapohos Tibor wrote:

> Thanks for your reply, Antonio.
>
> --- On Wed, 1/6/10, Antonio Perez wrote:
>
>> What's the output of:
>> fdisk -lu /dev/sda
>
> I will look at it tonight. Thanks.

Good.

>> However, the way you are making partitions is quite complex
>> and the needed space could be somewhere else.

You did "mdadm -C /dev/md0 -v -e 0 -l 1 -n 2 /dev/sda1 /dev/sdb1"

Which is the usual way to set the /boot partition.

However, you did repartition md0, while still expecting to boot from md0.

That is "odd" to me. Just make a FS in md0 and use it as /boot (and root).

> Does this mean that this partitioning is not the right way to proceed?
> should I create all the partitions first and build them into so many
> /dev/md[01234] devices separately maybe?

I could tell you what seems to be a reasonable setup IMO:
200Mb --> {sda1,sdb1} --> md1 --> RAID1 --> use as /boot
5Gb --> {sda2,sdb2} --> md2 --> RAID1 --> use as /
2GB --> independent sda3, sdb3 --> use as swap

Use the rest as needed. One option is:
rest --> {sda3,sdb3} --> md3 --> RAID1 --> LVM --> crete as many individual
disks as needed/wanted with lvcreate.

Having only two disks means to use only RAID1. RAID0 is too risky IMO as any
disk failure means a complete loss of data.

>> Please, stop repeating your question everywhere.

> I am very sorry if I upset the community. I did not think that linux-raid,
> grub-devel and help-grub lists would represent the same experts. In fact,
> based on what I see on the GRUB sites is, that grub-devel experts are
> different from more like "legacy" supporting help-grub ones. Maybe a bit
> of clarification would help keeping the developers/maintainers email
> intake a bit lower. So I must ask: which one is the most appropriate for
> my question?

As I am not an expert in any sense of the word, I really can't tell you
where to post. I am a common user which is trying to help you. But, in this
list alone, there are three post from you with exactly the same content and
question. If you are not getting any feedback, perhaps you should re-think
of a different way to make your questions. And, please, allow more time for
an answer.
Thanks for your cooperation.

--
Antonio Perez

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Thank you Antonio!

--- On Wed, 1/6/10, Antonio Perez wrote:

> However, you did repartition md0, while still expecting to
> boot from md0.
>=20
> That is "odd" to me. Just make a FS in md0 and use it as>
> /boot (and root).

I've been, and still am, puzzled by the efficiency of the calculations =
involved in setting up RAID, one way or the other. And what I refer to,=
goes down to number of clocks needed to perform the writing. As I was =
wondering down this path, I figured that the way I set it up would be t=
he most efficient. Of course, I do not know enough about the actual ker=
nel/raid code to be able to tell for sure. My intuition dictated it thi=
s way.

> > should I create all the partitions first and build
> > them into so many /dev/md[01234] devices separately maybe?
>
> I could tell you what seems to be a reasonable setup IMO:
>   =A0 200Mb --> {sda1,sdb1} --> md1 --> RAID1 --> use as /boot
>   =A0 5Gb  =A0--> {sda2,sdb2} --> md2 --> RAID1 --> use as /
>   =A0 ...

...and that's exactly what is also on my list, and your suggestion e=
nforces my thoughts, thanks. Nevertheless, it still does not explain to=
me whether multiple md devices with more underlying disk partitions or=
more md partitions with fewer underlying disk partitions are the way t=
o go. To be absolutely clear, is the

/dev/sd[ab]1 =3D> /dev/md0 =3D> /dev/md0p[123...] [RAID1]
/dev/sd[ab]2 =3D> /dev/md1 =3D> /dev/md1p[123...] [RAID0]

or the

/dev/sd[ab][123...] =3D> /dev/md[012...] [RAID1,RAID0]

setup the most stable, efficient and robust solution? The way I see it,=
there shall be a difference somewhere, somehow, right? About the first=
one, thanks to Michael E. pointing it out, we already know that one ha=
s to take care about choosing the patritions.

=46urthermore, I think that the gain in speed of RAID0 would start vani=
shing if disk access is required on the RAID1 side all at the same time=
In fact, the whole system's efficiency would degrade. The only way =
out would be to use 4 disks as

/dev/sd[ab]1 =3D> /dev/md0 =3D> /dev/md0p[123...] [RAID1]
/dev/sd[cd]1 =3D> /dev/md1 =3D> /dev/md1p[123...] [RAID0]

Right? Keep in mind that I am not after what RAID10 can offer. I can af=
ford giving up the data safety on the RAID0 side in exchange for speed.

> Having only two disks means to use only RAID1. RAID0 is too
> risky IMO as any disk failure means a complete loss of data.

I will be developing a numerical electromagnetics code, that will do qu=
ite some number crunching and dumping with 4xTesla C1060 GPUs. I will n=
eed the RAID0 for that purpose, for a scratch space, if you will. So it=
is a must have for me.

> I am a common user which is trying to help you.

Thanks, it's highly appreciated.

> But, in this list alone, there are three post from you with exactly t=
he
> same content and question.=20

That's really odd. I would never do that intentionally. As I wrote earl=
ier, I did submit it to 3 lists, help-grub, grub-devel and linux-raid, =
but not to the same one 3 times. The subject is related to both RAID an=
d GRUB, which is why I sent it 3 ways. My apologies again to all those =
who got it in 3 copies, whether by being subscribed to all 3 lists, or =
by a misterious triple submission of mine.=20

> If you are not getting any feedback, perhaps you should re-think=20
> of a different way to make your questions. And, please,
> allow more time for an answer.

I am a patient man, so not getting answers would not drive me to re-pos=
t the same message again.

> Thanks for your cooperation.

Absolutely. And I thank to all for your help.

All the best,
Tibor

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Lapohos Tibor wrote:

[...]
>> > should I create all the partitions first and build
>> > them into so many /dev/md[01234] devices separately maybe?

>> I could tell you what seems to be a reasonable setup IMO:
>> 200Mb --> {sda1,sdb1} --> md1 --> RAID1 --> use as /boot
>> 5Gb --> {sda2,sdb2} --> md2 --> RAID1 --> use as /
>> ...
>
> ....and that's exactly what is also on my list, and your suggestion
> enforces my thoughts, thanks. Nevertheless, it still does not explain to
> me whether multiple md devices with more underlying disk partitions or
> more md partitions with fewer underlying disk partitions are the way to
> go. To be absolutely clear, is the
>
> /dev/sd[ab]1 => /dev/md0 => /dev/md0p[123...] [RAID1]
> /dev/sd[ab]2 => /dev/md1 => /dev/md1p[123...] [RAID0]

I never meant to say to re-partition the md device!!
Why is that needed in a small 200 MByte partition?
I really like the a KISS approach.

> or the

> /dev/sd[ab][123...] => /dev/md[012...] [RAID1,RAID0]

> setup the most stable, efficient and robust solution? The way I see it,
> there shall be a difference somewhere, somehow, right? About the first
> one, thanks to Michael E. pointing it out, we already know that one has to
> take care about choosing the patritions.

I can't say what differences or efficiency is there, but it doesn't work!!
So, what does it matter?
Boot your system with the SIMPLE two RAID I stated above, and then, play as
you wish with as many raid levels and sub-partitions as you want with the
*rest* of the disks.

> Furthermore, I think that the gain in speed of RAID0 would start vanishing
> if disk access is required on the RAID1 side all at the same time.

> In fact, the whole system's efficiency would degrade. The only way out
> would be to use 4 disks as
>
> /dev/sd[ab]1 => /dev/md0 => /dev/md0p[123...] [RAID1]
> /dev/sd[cd]1 => /dev/md1 => /dev/md1p[123...] [RAID0]
>
> Right? Keep in mind that I am not after what RAID10 can offer. I can
> afford giving up the data safety on the RAID0 side in exchange for speed.

Yes, right. Do you have four disks?

>> Having only two disks means to use only RAID1. RAID0 is too
>> risky IMO as any disk failure means a complete loss of data.

> I will be developing a numerical electromagnetics code, that will do quite
> some number crunching and dumping with 4xTesla C1060 GPUs. I will need the
> RAID0 for that purpose, for a scratch space, if you will. So it is a must
> have for me.

I was just warning about a possible loss of data, if that is not an issue in
your system, do set up as many RAID0 as you need.

--
Antonio Perez

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Thanks for all your help again.

Things worked out nicely, and they left me wondering about what had really happened.

Trial #1:
I wiped out the disks, repartitioned them using fdisk starting the first partition at the 2nd cylinder to obtain sd[ab][12], and made md0 = sd[ab]1 and md1 = sd[ab]2. Installed GRUB2 without a problem.

Trial #2:
Then I repeated this very same thing, only this time I further partitioned the md[01] drives. The GRUB installation worked, again. The only problem was that before needing the root partition to become available the kernel would not detect the RAID partititons, only the arrays. So md0p1 was not available to be mounted, and this way I would need an initrd in the process.

Trial #3!!
I repeated Trial #1, by starting the firsst partition at the first cylinder. It worked! Therefore, I'm suspecting that cfdisk could have been the culprit the first time. How, I don't know.

Thanks again for all the attention you gave me,
Tibor

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

Lapohos Tibor wrote:

> Thanks for all your help again.

YW, all is well that ends well.

[...]
> ....How, I don't know.

Did you check the "fdisk -lu /dev/sda" output?

> Thanks again for all the attention you gave me,

YW, and good luck with your "numerical electromagnetics code".

--
Antonio Perez

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

> > ....How, I don't know.
>
> Did you check the "fdisk -lu /dev/sda" output?
>
Yes. 63, so I still went on and repartitioned as described in Trial #[123].

I'm thinking that it could have been something left behind by the Intel Matrix RAID volume management... Maybe Neil Brown could give an explanation to this. I also noticed that if one re-partitions the exact same way, the RAID superblock information becomes available again, that is, partitioning or disassembling a RAID device does not threcken those records off the disks. Therefore, even if one does not create a partitioned md array, it will be, on its own! Actually it makes sense, since partitioning does not involve alteration of records throughout the disks... I just thought it would be interesting to share all this.

In the near future I will describe my story in a single post, maybe it will inspire some developers to address certain points in their documentation work.

Thanks again to All,
Tibor

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html