WAR_STORIES: bash goes recursive

Apr 5, 2018   #Linux  #core  #gdb 

One day you are trying to copy a file to a remote host, but something goes terribly wrong:

$ scp /tmp/foo user@example.com:
lost connection

In fact, after you run scp it just hangs there for good 5 seconds, and then bails out. Meanwhile, ssh for this user works as expected - you can log in to that host, move around, and the host looks perfectly healthy. Oh well, what can it be? There is nothing to look at on the local host, but logging on the remote host we see:

# dmesg -T | tail -1
[Thu Apr  5 08:07:38 2018] bash[25457]: segfault at 7ffdd5895e88 ip 0000000000429f6a sp 00007ffdd5895e80 error 6 in bash[400000+f2000]

Yay, bash has a bad time. Did you just discover a new bug in bash? Let’s find out!

echo '/tmp/core.%e.%p' | tee /proc/sys/kernel/core_pattern 
/tmp/core.%e.%p

And add a couple of lines to /etc/security/limits.conf:

*       soft    core    unlimited
*       hard    core    unlimited

On a more modern system I would use coredumpctl gdb , but this particular host is old school:

gdb /bin/bash --core=/tmp/core.bash.26600
...
Core was generated by `bash -c scp -t .'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000429f6a in yyparse ()
...
bt

Boy, that is a big stacktrace. Fork bomb? Recursion? But why doesn’t it happen for a normal login? After installing debug symbols it becomes obvious that something goes wrong in .bashrc and it just blows up. Looking inside this file reveals that somehow home directory of this user was misconfigured, and _.bashprofile was renamed to_.bashrc, so now it tries to recursively call itself:

$ cat .bashrc 
# .bash_profile
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

When poor little scp spawns a remote shell, bash goes into infinite recursion and segfaults. Turns out, it was not a bash problem at all. On the second thought, the diagnostics could have been better, and I would rather have some sensible error message instead of segfault.