rmnl — remove new line characters with tr, awk, perl, sed or c/c++ 

How to remove new lines from files or pipe streams under Linux? This post contains simple examples that show how to use common Linux shell tools such as tr, awk/gawk, perl, sed and many others to delete new line characters. C and C++ source codes are also provided. They can be compiled into a binary tool that removes new lines. To get started, here is an example text file: days.txt. Lets have a look at its content by running the following command from shell.

cat days.txt ©2007-2008 dsplabs.com.au

The output is shown below.

Mon
Tue
Wed
Thu
Fri
Sat
Sun

If the new lines were removed from days.txt then its content would look like this:

MonTueWedThuFriSatSun

If instead of simply removing the new line characters they were replaced with spaces then the output would look as follows.

Mon Tue Wed Thu Fri Sat Sun

Below are simple examples of two types: one to remove new lines and the other to replace them with spaces. The outputs of these examples are as shown above.

tr — remove new line characters with tr

Either of the following commands can be used to delete new lines using the tr utility.

tr -d '\n' < days.txt
cat days.txt | tr -d '\n'

While the new line characters can be replaced with spaces using the tr program as follows.

tr '\n' ' ' < days.txt
cat days.txt | tr '\n' ' '

awk — remove new line characters with awk or gawk

Either of the following commands can be used to delete new lines using awk or gawk.

awk '{ printf "%s", $0 }' days.txt
cat days.txt | awk '{ printf "%s", $0 }'

While the new line characters can be replaced with spaces using either of the following commands.

awk '{ printf "%s ", $0 }' days.txt
cat days.txt | awk '{ printf "%s ", $0 }' 

Please be careful when using printf's: never pass any unvalidated inputs to printf as a format string as it is a security risk. That is, use: { printf "%s ", $0 } but not: { printf $0 }.

perl — remove new line characters with perl

Either of the following commands can be used to delete new lines using perl.

perl -e 'while (<>) { chomp; print; }; exit;' days.txt
cat days.txt | perl -e 'while (<>) { chomp; print; }; exit;' 

While the new line characters can be replaced with spaces using either of the following commands.

perl -e 'while (<>) { chomp; print; print " "; }; exit;' days.txt
cat days.txt | perl -e 'while (<>) { chomp; print; print " "; }; exit;' 

Here are some more perl examples:

perl -p -e 's/\s+$/ /g' days.txt
cat days.txt | perl -p -e 's/\s+$/ /g'

The above regex approach is actually much simpler and neater! :D

sed — remove new line characters with sed

You could also use sed to remove new lines. The solution using sed is not very readable but it works. Use either of the following commands.

sed ':a;N;$!ba;s/\n//g' days.txt
cat days.txt | sed ':a;N;$!ba;s/\n//g' 

Or, to replace the new line characters with spaces use either of the following commands.

sed ':a;N;$!ba;s/\n/ /g' days.txt
cat days.txt | sed ':a;N;$!ba;s/\n/ /g' 

James of crypto.dsplabs.com.au suggested this, easier to read, sed solution on the Linux Blog Forums:

sed '{:q;N;s/\n//g;t q}' days.txt

C or C++ — remove new line characters with a C or C++ based binary

If you fancy none of the above approaches then create your own binary tool for this task using for example C or C++. Here is an example C source code: rmnl.c also courtesy of James.

#include <stdio.h>
 
int main(int argc, char *argv[]) {
    int k; // using int here and not char since EOF is an int
    while((k = getchar()) != EOF)
        if(k != 0x0D && k != 0x0A) putchar(k);
//        else putchar(' '); // replace newlines with spaces
    putchar(0x0A);
    return 0;
}

While my attempt at writing a new line removal tool in C++, rmnl.cpp, is listed below.

#include <iostream>
using namespace std;
 
int main() {
    string line;
    while( getline(cin, line) && !cin.eof() ) cout << line; // delete newlines
//  while( getline(cin, line) && !cin.eof() ) cout << line << ' '; // replace n with spaces
    cout << endl;
    return 0;
}

Another way in C++ is to use stream iterators like so:

#include <iostream>
#include <iterator>
 
int main () {
    std::remove_copy( std::istream_iterator<char>( std::cin ),
                      std::istream_iterator<char>(),
                      std::ostream_iterator<char>( std::cout ),
                      '\n' );
}

To compile the above sources with GNU gcc or g++ compiler use the following commands, respectively.

gcc rmnl.c -o rmnl
g++ rmnl.cpp -o rmnl

Either of the following commands can be used to run the newly created rmnl binary assuming that rmnl is in your path.

cat days.txt | rmnl
rmnl < days.txt

If rmnl binary in not in your path variable, but it is in your current directory, then replace rmnl above with ./rmnl like so:

cat days.txt | ./rmnl
./rmnl < days.txt

Here is the compiled binary: rmnl and some information about it:

$>file rmnl
        rmnl: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
              dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

$>ldd rmnl
        linux-gate.so.1 =>  (0x0041d000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00b21000)
        libm.so.6 => /lib/libm.so.6 (0x0086c000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x009ec000)
        libc.so.6 => /lib/libc.so.6 (0x0072a000)
        /lib/ld-linux.so.2 (0x0070d000)

Other approaches for removal of new line characters

The following are thanks to reddit.com readers.

Replacing new lines with spaces using xargs:

xargs echo < days.txt
cat days.txt | xargs echo
cat days.txt | xargs echo -n

Removing new lines using Haskell, f.e. with the Glasgow Haskell Compiler, ghci:

cat days.txt | ghci -e "interact (concat . lines)"

Removing new lines or substituting them with spaces using sam text editor:

printf ", x/.*/nq" | sam -d days.txt 2>/dev/null
(echo ', x/\n/ c/ /'; echo ', p'; echo q) | sam -d days.txt 2>/dev/null

Or its stream version ssam:

ssam -n ', x/.*/ p' < days.txt 2>/dev/null
ssam ', x/\n/ c/ /' < days.txt 2>/dev/null

Another approach is to use GNU Bash like so:

while read; do echo -n "$REPLY "; done < days.txt

Or even simpler:

echo -n `cat days.txt`

Or, Python like this:

python -c 'import sys; print sys.stdin.read().replace("\n", " ")' < days.txt

References

Enjoy!


Did you find the above information useful and interesting? If so, please support this site by using the blog directory links at the bottom of this page. Thanks for your support!

If you have any Linux related problems or questions then please feel free to post them on our Linux Forums: http://linux.dsplabs.com.au/forums.




VPS Hosting Referral Code DZZCC3

Add me to Technorati Favorites Vote for me on Blog Catalog

30 Responses to “rmnl — remove new line characters with tr, awk, perl, sed or c/c++”

  1. Arjun Says:

    Awesome! I was pulling out my hair for quite some time trying to do this with sed, thanks for the post!

  2. tatwright Says:

    In perl you could use

    perl -p -e "chomp;"

  3. rmnl — remove new line characters with tr, awk, perl, sed or C/C++ : Linux T&T Says:

    […] Linux Blog » rmnl — remove new line characters with tr, awk, perl, sed or C/C++ […]

  4. guillem Says:

    With this code

    sed ':a;N;$!ba;s/\n/ /g'

    You saved my life. Many thanks :)

    I was wandering why is not as easy with sed to delete two consequitive lines as:

    sed 's|firs-line\second\line||g'

    Cheers from catalonia ;)

  5. Reemplaça múltiples linies de text amb el sed! « alliberats.org Says:

    […] a aquesta web vaig trobar la solució. Ja que el sed no reconeix la sintaxi obia que seria se […]

  6. How to make a HTTP Requst from command line « Identity Management , SOA, Testing, Monitoring Says:

    […] http://linux.dsplabs.com.au/rmnl-http://linux.dsplabs.com.au/rmnl-remove-new-line-characters-tr-awk-perl-sed-c-cpp-bash-python-xargs-ghc-ghci-haskell-sam-ssam-p65/remove-new-line-characters-tr-awk-perl-sed-c-cpp-bash-python-xargs-ghc-ghci-haskell-sam-ssam-p65/ […]

  7. Search & Replace Strings Considering the whole file (non line-based) using ’sed’ command « Saminda Wijeratne’s Weblog Says:

    […] so anyway what you have to do is pretty simple. either you have to format your input to remove the new line characters and feed that as the input and somehow reintroduce the newline character after sed was applied, or feed the whole input to sed in a non-line-based form. Anyway found this on the net. http://linux.dsplabs.com.au/rmnl-remove-new-line-characters-tr-awk-perl-sed-c-cpp-bash-python-xargs…. […]

  8. rofrol's status on Monday, 19-Oct-09 12:31:06 UTC - Identi.ca Says:

    […] http://linux.dsplabs.com.au/rmnl-remove-new-line-characters-tr-awk-perl-sed-c-cpp-bash-python-xargs… a few seconds ago from web […]

  9. i bashed my ls « RixBuntu Says:

    […] thanks to Linux Blog for the tutorial on removing new line characters with a variety of tools (tr, awk, perl, sed, […]

  10. Omkar Says:

    Awesome man! And this is how powerful linux is. Something like this in windows is almost impossible!

  11. Lietuva Says:

    Thank you very much for this article.

  12. Amanda Says:

    Wow! So many options to chose from, great reference!

  13. [SOLVED] simple sed question Says:

    […] http://linux.dsplabs.com.au/rmnl-rem…-sam-ssam-p65/ […]

  14. I need a shell script to delete files wich location should be read from a log file Says:

    […] list-tools Hi, I found this blog informative… http://linux.dsplabs.com.au/rmnl-rem…-sam-ssam-p65/ […]

  15. Reemplaça múltiples linies de text amb el sed! | GNULinux.cat Says:

    […] a aquesta web vaig trobar la solució. Ja que el sed no reconeix la sintaxi obia que seria se […]

  16. elmimmo Says:

    All this commands output to stdin, but how do I overwrite the original file with the new ouput?

  17. Aidas Kasparas Says:

    If you need to change newlines to spaces (or any other symbol) in perl there is an option

    perl -pl40 -e "

  18. Sayid Munawar Says:

    thanks !

    saved my day

  19. malef Says:

    thanks a lot! I have been looking for these sed solutions.

  20. aa Says:

    Watch out for cr+lf, as sometimes created with windows programs. http://en.wikipedia.org/wiki/Newline

  21. joewalp Says:

    I found the awk solution to be useful because it works when tailing a logfile, as in:
    tail -f -n 100000 misc.log | sed –unbuffered -n -e 's/.* APPEND finalize \(.\)/\1/p' | awk '{ printf "%s", $0 }'

  22. nilperlinux Says:

    thanks for the idea

  23. antoniomu Says:

    its a very usefull article

  24. Scott Severtson Says:

    Paste, a specialized line-joining utility available on most Unixes:
    # Without spaces
    paste -sd " days.txt
    # With spaces
    paste -sd ' ' days.txt

    An AWK alternative syntax that I prefer:
    # Without spaces
    awk 'BEGIN{ORS=""} {print}' days.txt
    # With spaces
    awk 'BEGIN{ORS=" "} {print}' days.txt
    # Skip blank lines, with spaces
    awk 'BEGIN{ORS=" "} $0 != "" {print}' days.txt

  25. Greg Benison Says:

    Sometimes you don't want to remove *all* of the newlines - for example to keep paragraphs intact. Here's a way to do that:

    http://gcbenison.wordpress.com/2011/07/03/a-program-to-intelligently-remove-carriage-returns-so-you-can-paste-text-without-having-it-look-awful/

  26. Vault»Blog Archive » All I must do is find it Says:

    […] second thing I wanted to link was an article posted on the Linux Blog related to removing new lines characters from text files.  The post mentions how to due this using tools like tr, awk, perl, sed, and C.  […]

  27. Cut - Sed - Awk | Pearltrees Says:

    […] How to remove new lines from files or pipe streams under Linux? This post contains simple examples that show how to use common Linux shell tools such as tr , awk/gawk , perl , sed and many others to delete new line characters. C and C++ source codes are also provided. Linux Blog » rmnl — remove new line characters with tr, awk, perl, sed or c/c++ […]

  28. Replacing new lines with nulls in bash | ProgClub Says:

    […] found a whole article on various ways to replace text on *nix: rmnl — remove new line characters with tr, awk, perl, sed or c/c++. This entry was posted in Programming, Sys Admin and tagged bash, new line, null, replace, tr, […]

  29. McUsr Says:

    Hello, I was googling this problem.

    I had the problems with runs of newlines interspersed with paragraphs of text, and I wanted to keep the paragraphs separated.

    I finally ended up with cat -s, which does that trick. I thought it worth mentioning, as that operation, seems to be utterly tricky with regular expressions, and tr, since tr is greedy.

  30. JC Says:

    Curiously, this does not work with the version of sed that ships with OSX (but works correctly under GNU sed 4.2.2)

    macmini:tmp jcw$ sed 'l' days.txt
    Mon$
    Mon
    Tue$
    Tue
    Wed$
    Wed
    Thu$
    Thu
    Fri$
    Fri
    Sat$
    Sat
    Sun$
    Sun

    This is as we expect, each line is terminated with a \n

    macmini:tmp jcw$ sed ':a;N;$!ba;l' days.txt
    Mon
    Tue
    Wed
    Thu
    Fri
    Sat
    Sun

    And this is as we expect.

    macmini:tmp jcw$ sed ':a;N;$!ba;s/\n/ /g' days.txt
    Mon
    Tue
    Wed
    Thu
    Fri
    Sat
    Sun

    This is NOT what we expect.

    macmini:tmp jcw$ sed ':a;N;$!ba;s/Mon/XXX/g' days.txt
    Mon
    Tue
    Wed
    Thu
    Fri
    Sat
    Sun

    This seems to make it pretty evident the regex isn't being applied at all.

    Of course, I'm no sed expert, but BSD sed clearly behaves differently (if not incorrectly) from GNU sed.

Leave a Reply