sed - using the same file for input and output

sed - using the same file for input and output

am 09.10.2007 17:40:25 von hyperboogie

Hello all

I'm trying to edit a file with sed in the following way:

sed 's/OldText/NewText/g' < myfile > myfile

using the same file for output as the input actually deletes the
content of myfile

the only way I do this is like so:

sed 's/OldText/NewText/g' < oldfile > newfile
cat newfile > oldfile

is there a way around this (using only sed)?
Thanks

Re: sed - using the same file for input and output

am 09.10.2007 18:11:49 von OldSchool

On Oct 9, 11:40 am, hyperboogie wrote:
> Hello all
>
> I'm trying to edit a file with sed in the following way:
>
> sed 's/OldText/NewText/g' < myfile > myfile
>
> using the same file for output as the input actually deletes the
> content of myfile
>
> the only way I do this is like so:
>
> sed 's/OldText/NewText/g' < oldfile > newfile
> cat newfile > oldfile
>
> is there a way around this (using only sed)?
> Thanks

gnu sed has the "-i" or --in-place option which does this. others may
as well.

Re: sed - using the same file for input and output

am 09.10.2007 18:39:36 von Icarus Sparry

On Tue, 09 Oct 2007 15:40:25 +0000, hyperboogie wrote:

> Hello all
>
> I'm trying to edit a file with sed in the following way:
>
> sed 's/OldText/NewText/g' < myfile > myfile
>
> using the same file for output as the input actually deletes the content
> of myfile
>
> the only way I do this is like so:
>
> sed 's/OldText/NewText/g' < oldfile > newfile cat newfile > oldfile
>
> is there a way around this (using only sed)? Thanks

As others have pointed out GNU sed has a "-i" option.

The problem is that it is the shell that has opened stdout and stdin for
you long before the sed is started.

You can do

( rm -f myfile ; sed 's/../../' > myfile) < myfile

which works as follows.
A subshell is started with its standard input connected to myfile.
The subshell runs the rm command (it happens to have its stdin connected
to myfile, but this doesn't matter as rm does not read any input).
A new "myfile" is created and attached to stdout of sed. Sed then has the
old myfile on its stdin, and the new one on stdout, and it can then do
its work.

Note that if the processes are interrupted then you will probably have
lost the contents of "myfile".

Re: sed - using the same file for input and output

am 09.10.2007 18:59:47 von Stephane CHAZELAS

2007-10-09, 15:40(-00), hyperboogie:
> Hello all
>
> I'm trying to edit a file with sed in the following way:
>
> sed 's/OldText/NewText/g' < myfile > myfile
>
> using the same file for output as the input actually deletes the
> content of myfile
>
> the only way I do this is like so:
>
> sed 's/OldText/NewText/g' < oldfile > newfile
> cat newfile > oldfile
>
> is there a way around this (using only sed)?
[...]

If the replacement text is exactly the same size as the replaced
text, you can do:

sed 's/OldText/NewText/g' < oldfile 1<> oldfile

FreeBSD and GNU seds have the -i option.

For portability, best is probably to use perl instead:

perl -pi -e 's/OldText/NewText/g' -- oldfile

or ed or ex, though those don't work in a stream fashion.

--
Stéphane

Re: sed - using the same file for input and output

am 09.10.2007 23:54:00 von Jens Binnewies

Stephane CHAZELAS wrote:

> If the replacement text is exactly the same size as the replaced
> text, you can do:
>
> sed 's/OldText/NewText/g' < oldfile 1<> oldfile
It works also if the replacement string is larger...
can you explain your above statement?



--
Jens

Re: sed - using the same file for input and output

am 10.10.2007 00:24:42 von Stephane CHAZELAS

2007-10-09, 23:54(+02), Jens Binnewies:
> Stephane CHAZELAS wrote:
>
>> If the replacement text is exactly the same size as the replaced
>> text, you can do:
>>
>> sed 's/OldText/NewText/g' < oldfile 1<> oldfile
> It works also if the replacement string is larger...
> can you explain your above statement?
[...]

If it's larger, it may work but will generally not.

In the above, sed reads and writes from the same file. sed
generally uses stdio or any sort of buffered IO. It will read a
block of data, process the lines in that data, and the converted
lines will be written as a block.

If the oldfile is less than one block size (depends on the
system), then it will be OK. If not, then it won't because when
the first block is written, it will overwrite the next block to
be read.

Example, let's say the block size is 4 and you want to replace
"a"'s with "ab"'s in a file containing "a\na\na\na\na\na\n".
sed will read one block "a\na\n", process those two lines, and
thus write "ab\nab\n" to stdout. Therefore, the file will now
contain:

ab\nab\na\na\na\n, then it will read the second block of 4
bytes: "b\na\n"... you see what I mean?

--
Stéphane

Re: sed - using the same file for input and output

am 10.10.2007 18:21:10 von hyperboogie

Thank you all

I think I'll stick to the perl workaround ...
I was hoping there was a "standard" sed way of doing this
oh well ...

Thanks anyhow
:-)

Re: sed - using the same file for input and output

am 10.10.2007 22:46:48 von Matej Cepl

On 2007-10-10, 16:21 GMT, hyperboogie wrote:
> I was hoping there was a "standard" sed way of doing this
> oh well ...

How much standard you need? -i parameter of (at least) GNU Sed is
not good enough, or did I miss some conditions you need to
satisfy?

Matěj

Re: sed - using the same file for input and output

am 10.10.2007 23:18:12 von cfajohnson

On 2007-10-10, Matej Cepl wrote:
>
>
> On 2007-10-10, 16:21 GMT, hyperboogie wrote:
>> I was hoping there was a "standard" sed way of doing this
>> oh well ...
>
> How much standard you need? -i parameter of (at least) GNU Sed is
> not good enough, or did I miss some conditions you need to
> satisfy?

The -i option is not standard. AFAIK, only the GNU and FreeBSD
versions have it.

What's wrong with the traditional UNIX method of redirecting to a
different file and moving or copying it back over the original
(after making sure that it is correct)? That is what I recommend,
especially for someone who has to ask for help with the command.

--
Chris F.A. Johnson, author
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence

Re: sed - using the same file for input and output

am 11.10.2007 01:09:19 von Kenan Kalajdzic

Chris F.A. Johnson wrote:
> On 2007-10-10, Matej Cepl wrote:
>>
>>
>> On 2007-10-10, 16:21 GMT, hyperboogie wrote:
>>> I was hoping there was a "standard" sed way of doing this
>>> oh well ...
>>
>> How much standard you need? -i parameter of (at least) GNU Sed is
>> not good enough, or did I miss some conditions you need to
>> satisfy?
>
> The -i option is not standard. AFAIK, only the GNU and FreeBSD
> versions have it.

And neither performs real in-place editing. Each of them rather
uses a temporary file, which is renamed to match the original file
name. AFAIK, only perl does in-place editing with the -i option.

> What's wrong with the traditional UNIX method of redirecting to a
> different file and moving or copying it back over the original
> (after making sure that it is correct)? That is what I recommend,
> especially for someone who has to ask for help with the command.

As mentioned above, what sed does is essentially the same, so the
recommendation is fully justified.

--
Kenan Kalajdzic

Re: sed - using the same file for input and output

am 11.10.2007 10:34:52 von hyperboogie

>
>
> How much standard you need? -i parameter of (at least) GNU Sed is
> not good enough, or did I miss some conditions you need to
> satisfy?

as stated by others it's not standard ... for example, most of the
machines I work on run Solaris on which sed does not include the i
option.
As I said, I think I'll stick to the perl workaround, which I've use
on several occasions before.

ps. - St=E9phane and Icarus Sparry - your exlanations were very
interesting and illuminating
Thanks :-)

what is ptfs in AIX

am 11.10.2007 16:46:22 von jena.sk

On Oct 9, 9:39 pm, Icarus Sparry wrote:
> On Tue, 09 Oct 2007 15:40:25 +0000, hyperboogie wrote:
> > Hello all
>
> > I'm trying to edit a file with sed in the following way:
>
> > sed 's/OldText/NewText/g' < myfile > myfile
>
> > using the same file for output as the input actually deletes the content
> > of myfile
>
> > the only way I do this is like so:
>
> > sed 's/OldText/NewText/g' < oldfile > newfile cat newfile > oldfile
>
> > is there a way around this (using only sed)? Thanks
>
> As others have pointed out GNU sed has a "-i" option.
>
> The problem is that it is the shell that has opened stdout and stdin for
> you long before the sed is started.
>
> You can do
>
> ( rm -f myfile ; sed 's/../../' > myfile) < myfile
>
> which works as follows.
> A subshell is started with its standard input connected to myfile.
> The subshell runs the rm command (it happens to have its stdin connected
> to myfile, but this doesn't matter as rm does not read any input).
> A new "myfile" is created and attached to stdout of sed. Sed then has the
> old myfile on its stdin, and the new one on stdout, and it can then do
> its work.
>
> Note that if the processes are interrupted then you will probably have
> lost the contents of "myfile".

Re: sed - using the same file for input and output

am 15.10.2007 23:03:14 von Stephane CHAZELAS

2007-10-11, 01:09(+02), Kenan Kalajdzic:
> Chris F.A. Johnson wrote:
>> On 2007-10-10, Matej Cepl wrote:
>>>
>>>
>>> On 2007-10-10, 16:21 GMT, hyperboogie wrote:
>>>> I was hoping there was a "standard" sed way of doing this
>>>> oh well ...
>>>
>>> How much standard you need? -i parameter of (at least) GNU Sed is
>>> not good enough, or did I miss some conditions you need to
>>> satisfy?
>>
>> The -i option is not standard. AFAIK, only the GNU and FreeBSD
>> versions have it.
>
> And neither performs real in-place editing. Each of them rather
> uses a temporary file, which is renamed to match the original file
> name. AFAIK, only perl does in-place editing with the -i option.
[...]

No, perl is the same. It writes a new file by the same name as
the original one's. It does something similar as:

{ rm file; sed 's/.../.../g' > file; } < file

except that it also tries to dupplicate the file ownership and
permissions.

--
Stéphane

Re: sed - using the same file for input and output

am 16.10.2007 00:19:15 von Janis Papanagnou

Stephane CHAZELAS wrote:
> 2007-10-11, 01:09(+02), Kenan Kalajdzic:
>
>>Chris F.A. Johnson wrote:
>>
>>>On 2007-10-10, Matej Cepl wrote:
>>>
>>>>
>>>>On 2007-10-10, 16:21 GMT, hyperboogie wrote:
>>>>
>>>>>I was hoping there was a "standard" sed way of doing this
>>>>>oh well ...
>>>>
>>>>How much standard you need? -i parameter of (at least) GNU Sed is
>>>>not good enough, or did I miss some conditions you need to
>>>>satisfy?
>>>
>>> The -i option is not standard. AFAIK, only the GNU and FreeBSD
>>> versions have it.
>>
>>And neither performs real in-place editing. Each of them rather
>>uses a temporary file, which is renamed to match the original file
>>name. AFAIK, only perl does in-place editing with the -i option.
>
> [...]
>
> No, perl is the same. It writes a new file by the same name as
> the original one's. It does something similar as:
>
> { rm file; sed 's/.../.../g' > file; } < file

I hope that "similar" does not mean "as fatal as the above shell
construct is to a file" in case, e.g., the file system is nearly
full.

Janis

>
> except that it also tries to dupplicate the file ownership and
> permissions.
>

Re: sed - using the same file for input and output

am 16.10.2007 08:41:26 von Stephane CHAZELAS

2007-10-16, 00:19(+02), Janis Papanagnou:
[...]
>> { rm file && sed 's/.../.../g' > file; } < file
>
> I hope that "similar" does not mean "as fatal as the above shell
> construct is to a file" in case, e.g., the file system is nearly
> full.
[...]

/mnt$ df -k .
Filesystem 1K-blocks Used Available Use% Mounted on
/home/chazelas/a 487 15 447 4% /mnt
/mnt$ yes | head -200000 > a
/mnt$ wc a
200000 200000 400000 a
/mnt$ df -k .
Filesystem 1K-blocks Used Available Use% Mounted on
/home/chazelas/a 487 409 53 89% /mnt
/mnt$ perl -pi -e 's/./../g' a
/mnt$ wc a
17749 17750 53248 a

It's worse, there's not even a warning.

--
Stéphane