Multiple commands during an SSH inside an SSH session

by

I have a local machine which is supposed to make an SSH session to a remote master machine and then another inner SSH session from the master to each of some remote slaves, and then execute 2 commands i.e. to delete a specific directory and recreate it.

Note that the local machine has passwordless SSH to the master and the master has passwordless SSH to the slaves. Also all hostnames are known in .ssh/config of the local/master machines and the hostnames of the slaves are in slaves.txt locally and I read them from there.

So what I do and works is this:

username="ubuntu"
masterHostname="myMaster"
while read line
do

    #Remove previous folders and create new ones.
    ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="a581d0d6c0d7cbc4c8c0e5">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="83a7f6f0e6f1ede2eee6c3">[email protected]</a>$line "rm -rf Input Output Partition""
    ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="cfebbabcaabda1aea2aa8f">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="381c4d4b5d4a5659555d78">[email protected]</a>$line "mkdir -p EC2_WORKSPACE/$project Input Output Partition""


    #Update changed files...
    ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="183c6d6b7d6a7679757d58">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="6b4f1e180e19050a060e2b">[email protected]</a>$line "rsync --delete -avzh /EC2_NFS/$project/* EC2_WORKSPACE/$project""

done < slaves.txt

This cluster is on Amazon EC2 and I have noticed that there are 6 SSH sessions created at each iteration which induces a significant delay. I would like to combine these 3 commands into 1 to get fewer SSH connections. So I tried to combine the first 2 commands into 

ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f2d6878197809c939f97b2">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="1135646274637f707c7451">[email protected]</a>$line "rm -rf Input Output Partition && mkdir -p EC2_WORKSPACE/$project Input Output Partition""

But it doesn’t work as expected. It seems to execute the first one (rm -rf Input Output Partition) and then exits the session and goes on. What can I do?

Contents hide
Answers:
Method 1
Method 2
Method 3
Method 4

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Consider that && is a logical operator. It does not mean “also run this command” it means “run this command if the other succeeded”.

That means if the rm command fails (which will happen if any of the three directories don’t exist) then the mkdir won’t be executed. This does not sound like the behaviour you want; if the directories don’t exist, it’s probably fine to create them.

Use ;

The semicolon ; is used to separate commands. The commands are run sequentially, waiting for each before continuing onto the next, but their success or failure has no impact on each other.

Escape inner quotes

Quotes inside other quotes should be escaped, otherwise you’re creating an extra end point and start point. Your command:

ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="7551000610071b14181035">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="4367363026312d222e2603">[email protected]</a>$line "rm -rf Input Output Partition && mkdir -p EC2_WORKSPACE/$project Input Output Partition""

Becomes:

ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="9abeefe9ffe8f4fbf7ffda">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="381c4d4b5d4a5659555d78">[email protected]</a>$line "rm -rf Input Output Partition && mkdir -p EC2_WORKSPACE/$project Input OutputPartition""

Your current command, because of the lack of escaped quotes should be executing:

ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="587c2d2b3d2a3639353d18">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="8da9f8fee8ffe3ece0e8cd">[email protected]</a>$line "rm -rf Input Output Partition

if that succeeds:

mkdir -p EC2_WORKSPACE/$project Input Output Partition"" # runs on your local machine

You’ll notice the syntax highlighting shows the entire command as red on here, which means the whole command is the string being passed to ssh. Check your local machine; you may have the directories Input Output and Partition where you were running this.

Method 2

You can always define in your jumpbox Multiplexing in OpenSSH

Multiplexing is the ability to send more than one signal over a single
line or connection. With multiplexing, OpenSSH can re-use an existing
TCP connection for multiple concurrent SSH sessions rather than
creating a new one each time.

An advantage with SSH multiplexing is that the overhead of creating
new TCP connections is eliminated. The overall number of connections
that a machine may accept is a finite resource and the limit is more
noticeable on some machines than on others, and varies greatly
depending on both load and usage. There is also significant delay when
opening a new connection. Activities that repeatedly open new
connections can be significantly sped up using multiplexing.

For that do in /etc/ssh/ssh_config:

ControlMaster auto
ControlPath ~/.ssh/controlmasters/ssh_mux_%h_%p_%r
ControlPersist 30m

In this way, any consecutive connections made to the same server in the following 30 minutes will be done reusing the previous ssh connection.

You can also define it for a machine or group of machines. Taken from the link provided.

Host machine1
    HostName machine1.example.org
    ControlPath ~/.ssh/controlmasters/%<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="97e5d7">[email protected]</a>%h:%p
    ControlMaster auto
    ControlPersist 10m

Method 3

You can put all your commands into a separate script on your “master” server.

Master Script

#!/bin/bash
rm -rf "Input Output Partition"
mkdir -p "EC2_WORKSPACE/$project Input Output Partition"

Then in your ssh script call it like this:
SSH Script

username="ubuntu"
masterHostname="myMaster"
while read line
do
ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="36124345534458575b5376">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="87a3f2f4e2f5e9e6eae2c7">[email protected]</a>$line < /path/to/masterscript.sh"
ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="e5c1909680978b848880a5">[email protected]</a>$masterHostname "ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f3d7868096819d929e96b3">[email protected]</a>$line "rsync --delete -avzh /EC2_NFS/$project/* EC2_WORKSPACE/$project""
done < slaves.txt

OR
if all files must be on the initial machine you could do something like this:

script1

script2="/path/to/script2"
username="ubuntu"
while read line; do
cat $script2 | ssh -t -t <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f7d38284928599969a92b79b9e9992">[email protected]</a>
done < slaves.txt

script2

#!/bin/bash
rm -rf "Input Output Partition"
mkdir -p "EC2_WORKSPACE/$project Input Output Partition"
rsync --delete -avzh "/EC2_NFS/$project/* EC2_WORKSPACE/$project"

ssh script

script1="/path/to/script1"
username="ubuntu"
masterHostname="myMaster"
cat $script1 | ssh -n <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="81a5f4f2e4f3efe0ece4c1">[email protected]</a>$masterHostname

Method 4

Some time ago, I had occasion to use control sockets like the other answers recommend (this answer is essentially a combination of using control sockets like this answer and scripts like this answer).

The use case was a hack: The authorized_keys of the target user was overwritten periodically by a scheduled task and I wanted to quickly test things without going through red-tape needed to add something to that file. So I’d setup a while loop which added the key to that file as needed, run my test, and the cancel the loop. However, there would be a small window where the scheduled task would overwrite the file and my loop would still be sleeping. So, setting up a control socket at the start would let my script SSH later without problems:

#! /bin/bash -xe
. "${CONFIG_DIR}/scripts/setup-ssh.sh"

# Build and test
export TEST_LABEL="${_started_by}-${BUILD_TAG%-BUILD*}"
#...
xargs --arg-file test-list 
    --no-run-if-empty 
    --process-slot-var=NUM 
    --max-procs=${#SERVERS[@]} 
    --max-args="${BATCH_SIZE:-20}" 
    "${CONFIG_DIR}/scripts/run-test.sh"

Where setup-ssh.sh is:

export SSH_CONFIG="${CONFIG_DIR}/scripts/.ssh-config"
mapfile -t SERVERS < "${CONFIG_DIR}/scripts/hosts"

for SERVER in "${SERVERS[@]}"
do
    while ! ssh -F "${SSH_CONFIG}" "${SERVER}" -fnN; do sleep 1; done
    scp -F "${SSH_CONFIG}" "${CONFIG_DIR}/scripts/ssh-script.sh" "${SERVER}":"${TEST_LABEL}.sh"
done

And .ssh-config:

Host test-*
  User test
  StrictHostKeyChecking no
  ControlMaster auto
  ControlPath /tmp/ssh-%h-%p-%r

And run-test.sh:

mapfile -t TEST_SERVERS < "${CONFIG_DIR}/scripts/hosts"
ssh -F "${SSH_CONFIG}" "${TEST_SERVERS[$NUM]}" "./${TEST_LABEL}.sh"

The sequence goes like this:

  • The main script (shown first) sources setup-ssh.sh.
  • setup-ssh.sh busy-loops the servers until all of them have a control socket setup. The hosts file simply lists the server hostnames one per line.
  • Since the configuration specifying the control socket is only in ${CONFIG_DIR}/scripts/.ssh-config, unless I specify that file using -F, SSH connections won’t use it. So this allows me to use the control socket only where I need them using the F option.
  • The setup script also copies the test execution script to the servers. The execution script itself contains a bunch of commands, and because I copied the execution script, I don’t have to worry about an additional layer of quoting for SSH (and the additional cognitive overhead for figuring out what gets expanded when).
  • Then the main script uses xargs to distribute the workload over the servers, by starting new jobs as soon as running ones end.


All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

0 0 votes
Article Rating
Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x