Replace a long string with the sed command: Argument list too long error

I’m trying to execute this command

sed -i -e "s/BASE_64/$BASE_64/" FILE_NAME

where $BASE_64 is a base 64 representation of a file content.

sed gives me an error since the string is too long.

Argument list too long

How is it possible to avoid this error?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

You could always do (since you’re using GNU sed already (-i)):

sed -i -f - FILE_NAME << EOF
s/BASE_64/$BASE_64/g
EOF

-f - tells sed to read the sed script from stdin.

If you want to reuse the same script for several files, on Linux (and Linux only), with a shell like zsh, ksh and bash version up to 5.0 that implements here documents with temporary files (as opposed to pipes like dash or yash (or bash 5.1+) for relatively small heredocs and still with GNU sed, you could do:

find . -name '*.conf' -exec sed -i -f /dev/stdin {} + << EOF
s/BASE_64/$BASE_64/g
EOF

On Linux (and Linux and Cygwin only), /dev/stdin does not mean stdin in the same way - does. Instead, it’s a symlink to the file open on stdin, so each time sed opens it, it opens the file anew from the start. The above command would work OK on other systems (that have /dev/stdin) or with shells that implement here-documents with pipes but only if there are few enough conf files that sed is called only once. When called the second time, on non-Linux/Cygwin systems, like with -f -, /dev/stdin would appear empty because it has already been read by the first invocation.

busybox sed also supports -i in the same way as GNU sed does, but does not support -f -. So you’d want to use -f /dev/stdin there in any case. With FreeBSD sed, use:

sed -i '' -f /dev/stdin FILE_NAME << EOF
s/BASE_64/$BASE_64/g
EOF

Method 2

First, save the base64-encoded data in a file called, e.g., base64.txt.

For example:

base64 < originalfile > base64.txt

Then:

printf '%sn' '/BASE64/r base64.txt' 1 '/BASE64/d' w | ed FILENAME

This uses ed to search in FILENAME for a line containing the string BASE64, insert the contents of base64.txt after that line, go back to the first line, then search for the line with string BASE64 again and delete it. The w command in ed saves the modified file.

Method 3

Another option would be to replace sed with ed and store your commands in a file. For example, if you create ed_cmds with the following contents:

%s/BASE_64/<expanded variable>/g
w
q

you could then run

< ed_cmds ed FILE_NAME

and it would make the changes you wanted, so instead of setting $BASE_64 you’d create the ed command file.

Ed Explanation

• % means to apply the command to each line of the file
• s/pat1/pat2/g substitutes occurrences of pat1 with pat2 and g at the end makes it do it for every match on the line, not just the first
• w write the changes to disk
• q quit (which would happen when it got EOF anyway)

Of course, you could put your sed commands in a file and use -f as well, but if you’re doing that and you want to modify the file in place you might as well use ed instead of creating a temporary file and moving it as sed -i does.

Method 4

The size of the Base64 representation of the file ($BASE_64) is too long and exceeds maximum argument size. You should be able to see this limit for your system by running

getconf ARG_MAX

You have to increase the size of the ARG_MAX value.
But I think if the file is too big, then you will have to use a different approach to do this replacement. If a Python script would do it for you, i’d try it with that too.

Method 5

I ended up putting the sed instructions into a file

SEDCOMMANDS=`tempfile`

and called

sed -f "$SEDCOMMANDS" -- "$FILE_NAME"

That’s good if you don’t use sed -i. If you want to edit the file in place, follow https://unix.stackexchange.com/a/284188/149867 and put the equivalent ed instructions in a file, followed by w and q.

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0