This articleneeds additional citations forverification. Please helpimprove this article byadding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Newline" – news ·newspapers ·books ·scholar ·JSTOR(February 2016) (Learn how and when to remove this message) |
Anewline (frequently calledline ending,end of line (EOL),next line (NEL) orline break) is acontrol character or sequence of control characters incharacter encoding specifications such asASCII,EBCDIC,Unicode, etc. This character, or a sequence of characters, is used to signify the end of aline of text and the start of a new one.[1]
In the mid-1800s, long before the advent ofteleprinters and teletype machines,Morse code operators ortelegraphists invented and usedMorse code prosigns to encode white space text formatting in formal written text messages. In particular, theMorse prosignBT (mnemonic break text), represented by the concatenation of literal textual Morse codes "B" and "T" characters, sent without the normal inter-character spacing, is used in Morse code to encode and indicate anew line ornew section in a formal text message.
Later, in the age of modernteleprinters, standardized character set control codes were developed to aid in white space text formatting. ASCII was developed simultaneously by theInternational Organization for Standardization (ISO) and the American Standards Association (ASA), the latter being the predecessor organization toAmerican National Standards Institute (ANSI). During the period of 1963 to 1968, the ISO draft standards supported the use of eitherCR+LF orLF alone as a newline, while the ASA drafts supported onlyCR+LF.
The sequenceCR+LF was commonly used on many early computer systems that had adoptedTeletype machines—typically aTeletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line. The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in time to print the next character. Any character printed after aCR would often print as a smudge in the middle of the page while the print head was still moving the carriage back to the first position. "The solution was to make the newline two characters:CR to move the carriage to column one, andLF to move the paper up."[2] In fact, it was often necessary to sendextra padding characters—extraneous CRs or NULs—which are ignored but give the print head time to move to the left margin. Many early video displays also required multiple character times toscroll the display.
On such systems, applications had to talk directly to the Teletype machine and follow its conventions since the concept ofdevice drivers hiding such hardware details from the application was not yet well developed. Therefore, text was routinely composed to satisfy the needs of Teletype machines. Most minicomputer systems fromDEC used this convention.CP/M also used it in order to print on the same terminals that minicomputers used. From thereMS-DOS (1981) adoptedCP/M'sCR+LF in order to be compatible, and this convention was inherited by Microsoft's laterWindows operating system.
TheMultics operating system began development in 1964 and usedLF alone as its newline. Multics used a device driver to translate this character to whatever sequence a printer needed (including extrapadding characters), and the single byte was more convenient for programming. What seems like a more obvious choice—CR—was not used, asCR provided the useful function of overprinting one line with another to createboldface,underscore andstrikethrough effects. Perhaps more importantly, the use ofLF alone as a line terminator had already been incorporated into drafts of the eventualISO/IEC 646 standard.Unix followed the Multics practice, and laterUnix-like systems followed Unix. This created conflicts between Windows and Unix-likeoperating systems, whereby files composed on one operating system could not be properly formatted or interpreted by another operating system (for example aUNIX shell script written in a Windows text editor likeNotepad[3][4]).
The concepts ofcarriage return (CR) and line feed (LF) are closely associated and can be considered either separately or together. In the physical media oftypewriters andprinters, twoaxes of motion, "down" and "across", are needed to create a new line on thepage. Although the design of a machine (typewriter or printer) must consider them separately, the abstract logic of software can combine them together as one event. This is why a newline incharacter encoding can be defined asCR
andLF
combined into one (commonly calledCR+LF
orCRLF
).
Somecharacter sets provide a separate newline character code.EBCDIC, for example, provides anNL character code in addition to theCR andLF codes.Unicode, in addition to providing theASCIICR andLFcontrol codes, also provides a "next line" (NEL) control code, as well as control codes for "line separator" and "paragraph separator" markers. Unicode also contains printable characters for visually representing line feed ␊, carriage return ␍, and other C0 control codes (as well as a generic newline, ) in theControl Pictures block.
Operating system | Character encoding | Abbreviation | hex value | dec value | Escape sequence |
---|---|---|---|---|---|
Multics POSIX standard oriented systems:Unix andUnix-like systems (Linux,macOS,*BSD,AIX,Xenix, etc.),QNX 4+ Others:BeOS,Amiga,RISC OS, and others[5] | ASCII | LF | 0A | 10 | \n |
Windows,MS-DOS compatibles,Atari TOS,DECTOPS-10,RT-11,CP/M,MP/M,OS/2,Symbian OS,Palm OS,Amstrad CPC, and most other early non-Unix and non-IBM operating systems | CR LF | 0D 0A | 13 10 | \r\n | |
Commodore 64,Commodore 128,Acorn BBC,ZX Spectrum,TRS-80,Apple II,Oberon,classic Mac OS,HP Series 80, MITLisp Machine, andOS-9 | CR | 0D | 13 | \r | |
Acorn BBC[6] andRISC OS spooled text output[7] | LF CR | 0A 0D | 10 13 | \n\r | |
QNX pre-POSIX implementation (version < 4) | RS | 1E | 30 | \036 | |
Atari 8-bit computers | ATASCII | EOL | 9B | 155 | |
IBM mainframe systems, includingz/OS (OS/390) andIBM i (OS/400) | EBCDIC | NL | 15 | 21 | \025 |
ZX80 andZX81 (home computers fromSinclair Research Ltd) | ZX80/ZX81 proprietary encoding | 76 | 118 |
0x85
) is calledNEL (Next Line). EBCDIC also has control characters calledCR andLF, but the numerical value ofLF (0x25) differs from the one used by ASCII (0x0A). Additionally, some EBCDIC variants also useNL but assign a different numeric code to the character. However, those operating systems use arecord-based file system, which stores text files as one record per line. In most file formats, no line terminators are actually stored.#
that thus could not be used as the first character in a line. Some early line printers interpreted these characters directly in the records sent to them.Many communications protocols have some sort of new line convention. In particular, protocols published by theInternet Engineering Task Force (IETF) typically use the ASCII CRLF sequence.
In some older protocols, the new line may be followed by a checksum or parity character.
TheUnicode standard defines a number of characters that conforming applications should recognize as line terminators:[9]
LF: | Line Feed,U+000A |
VT: | Vertical Tab,U+000B |
FF: | Form Feed,U+000C |
CR: | Carriage Return,U+000D |
CR+LF: | CR (U+000D) followed byLF (U+000A) |
NEL: | Next Line,U+0085 |
LS: | Line Separator,U+2028 |
PS: | Paragraph Separator,U+2029 |
While it may seem overly complicated compared to an approach such as converting all line terminators to a single character (e.g.LF), because Unicode is designed to preserve all information when converting a text file from any existing encoding to Unicode and back (round-trip integrity), Unicode needs to make the same distinctions between line breaks made by other encodings. For instanceEBCDIC hasNL,CR, andLF characters, so all three have to also exist in Unicode.
Most newline characters and sequences are inASCII'sC0 controls (i.e. have Unicode code points up to0x1F). The three newline characters outside of this range—NEL,LS andPS—are often not recognized as newlines by software. For example:
Unicode includes someglyphs intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline:
To facilitate creatingportable programs,programming languages provide some abstractions to deal with the different types of newline sequences used in different environments.
TheC language provides theescape sequences\n
(newline) and\r
(carriage return). However, these are not required to be equivalent to the ASCIILF andCR control characters. The C standard only guarantees two traits:
\n
is transparently translated to the native newline sequence used by the system, which may be longer than one character. When reading in text mode, the native newline sequence is translated back to\n
. Inbinary mode, no translation is performed, and the internal representation produced by\n
is output directly.OnUnixoperating system platforms, where C originated, the native newline sequence is ASCIILF (0x0A), so\n
was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode is ano-op, and Unix has no notion of text mode or binary mode. This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms.
TheC standard library functionfgets() is best avoided in binary mode because any file not written with the Unix newline convention will be misread. Also, in text mode, any file not written with the system's native newline sequence (such as a file created on a Unix system, then copied to a Windows system) will be misread as well.
Another common problem is the use of\n
when communicating using an Internet protocol that mandates the use of ASCIICR+LF for ending lines. Writing\n
to a text mode stream works correctly on Windows systems, but produces onlyLF on Unix, and something completely different on more exotic systems. Using\r\n
in binary mode is slightly better.
Many languages, such asC++,Perl,[16] andHaskell provide the same interpretation of\n
as C. C++ has analternative input/output (I/O) model where the manipulatorstd::endl can be used to output a newline (and flushes the stream buffer).
Java,PHP,[17] andPython[18] provide the\r\n
sequence (for ASCIICR+LF). In contrast to C, these are guaranteed to represent the valuesU+000D andU+000A, respectively.
TheJava Class Libraryinput/output (I/O) methods do not transparently translate these into platform-dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any ofCR,LF, orCR+LF as a line terminator (seeBufferedReader.readLine()). TheSystem.lineSeparator() method can be used to retrieve the underlying line separator.
Example:
Stringeol=System.lineSeparator();StringlineColor="Color: Red"+eol;
Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file.[19]
Some languages have created specialvariables,constants, andsubroutines to facilitate newlines during program execution. In some languages such asPHP andPerl,double quotes are required to perform escape substitution for all escape sequences, including\n
and\r
. In PHP, to avoid portability problems, newline sequences should be issued using the PHP_EOL constant.[20]
Example inC#:
stringeol=Environment.NewLine;stringlineColor="Color: Red"+eol;stringeol2="\n";stringlineColor2="Color: Blue"+eol2;
The different newline conventions cause text files that have been transferred between systems of different types to be displayed incorrectly.
Text in files created with programs which are common onUnix-like orclassic Mac OS, appear as a single long line on most programs common toMS-DOS andMicrosoft Windows because these do not display a singleline feed
or a singlecarriage return
as a line break.
Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extraCR may be displayed as a second line break, as^M, or as<cr> at the end of each line.
Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file.
The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. For example, acompiler may fail with obscure syntax errors even though the source file looks correct when displayed on theconsole or in aneditor. Modern text editors generally recognize all flavours ofCR+LF newlines and allow users to convert between the different standards.Web browsers are usually also capable of displaying text files and websites which use different types of newlines.
Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save—often inconsistently.
Most textualInternetprotocols (includingHTTP,SMTP,FTP,IRC, and many others) mandate the use of ASCIICR+LF (\r\n
,0x0D 0x0A) on the protocol level, but recommend that tolerant applications recognize loneLF (\n
,0x0A) as well. Despite the dictated standard, many applications erroneously use theC newline escape sequence\n
(LF) instead of the correct combination of carriage return escape and newline escape sequences\r\n
(CR+LF) (see sectionNewline in programming languages above). This accidental use of the wrong escape sequences leads to problems when trying to communicate with systems adhering to the stricter interpretation of the standards instead of the suggested tolerant interpretation. One such intolerant system is theqmailmail transfer agent that actively refuses to accept messages from systems that send bareLF instead of the requiredCR+LF.[21]
The standard Internet Message Format[22] for email states: "CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body". Differences between SMTP implementations in how they treat bare LF and/or bare CR characters have led to SMTP spoofing attacks referred to as "SMTP smuggling".[23]
TheFile Transfer Protocol can automatically convert newlines in files being transferred betweensystems with different newline representations when the transfer is done in "ASCII mode". However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectivelycorrupting the file. FTP clients often employ someheuristics (for example, inspection offilename extensions) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.[24]
Text editors are often used for converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCIICR/LF conventions.
For example, the editorVim can make a file compatible with the Windows Notepad text editor. Within vim
:setfileformat=dos :wq
Editors can be unsuitable for converting larger files or bulk conversion of many files. For larger files (on Windows NT) the following command is often used:
D:\>TYPE unix_file| FIND /V""> dos_file
Special purpose programs to convert files between different newline conventions includeunix2dos anddos2unix,mac2unix andunix2mac,mac2dos anddos2mac, andflip.[25]Thetr command is available on virtually everyUnix-like system and can be used to perform arbitrary replacement operations on single characters. A DOS/Windows text file can be converted to Unix format by simply removing all ASCIICR characters with
$tr -d '\r' <inputfile >outputfile
or, if the text has onlyCR newlines, by converting allCR newlines toLF with
$tr '\r' '\n' <inputfile >outputfile
The same tasks are sometimes performed withawk,sed, or inPerl if the platform has a Perl interpreter:
$awk'{sub("$","\r\n"); printf("%s",$0);}'inputfile>outputfile# UNIX to DOS (adding CRs on Linux and BSD based OS that haven't GNU extensions)$awk'{gsub("\r",""); print;}'inputfile>outputfile# DOS to UNIX (removing CRs on Linux and BSD based OS that haven't GNU extensions)$sed-e's/$/\r/'inputfile>outputfile# UNIX to DOS (adding CRs on Linux based OS that use GNU extensions)$sed-e's/\r$//'inputfile>outputfile# DOS to UNIX (removing CRs on Linux based OS that use GNU extensions)$perl-pe's/\r?\n|\r/\r\n/g'inputfile>outputfile# Convert to DOS$perl-pe's/\r?\n|\r/\n/g'inputfile>outputfile# Convert to UNIX$perl-pe's/\r?\n|\r/\r/g'inputfile>outputfile# Convert to old Mac
Thefile command can identify the type of line endings:
$filemyfile.txt myfile.txt: ASCII English text, with CRLF line terminators
The Unixegrep (extended grep) command can be used to print filenames of Unix or DOS files (assuming Unix and DOS-style files only, no classic Mac OS-style files):
$egrep-L'\r\n'myfile.txt# show UNIX style file (LF terminated)$egrep-l'\r\n'myfile.txt# show DOS style file (CRLF terminated)
Other tools permit the user to visualise the EOL characters:
$od-amyfile.txt$cat-emyfile.txt$cat-vmyfile.txt$hexdump-cmyfile.txt
Two ways to view newlines, both of which areself-consistent, are that newlines eitherseparate lines or that theyterminate lines. If a newline is considered a separator, there will be no newline after the last line of a file. Some programs have problems processing the last line of a file if it is not terminated by a newline. On the other hand, programs that expect newline to be used as a separator will interpret a final newline as starting a new (empty) line. Conversely, if a newline is considered a terminator, all text lines including the last are expected to be terminated by a newline. If the final character sequence in a text file is not a newline, the final line of the file may be considered to be an improper or incomplete text line, or the file may be considered to be improperly truncated.
In text intended primarily to be read by humans using software which implements theword wrap feature, a newline character typically only needs to be stored if a line break is required independent of whether the next word would fit on the same line, such as betweenparagraphs and in vertical lists. Therefore, in the logic ofword processing and mosttext editors, newline is used as aparagraph break and is known as a "hard return", in contrast to "soft returns" which are dynamically created to implement word wrapping and are changeable with each display instance. In many applications a separatecontrol character called "manual line break" exists for forcing line breaks inside a single paragraph. Theglyph for the control character for a hard return is usually apilcrow (¶), and for the manual line break is usually a carriage return arrow (↵).
RI (U+008DREVERSE LINE FEED,[26]ISO/IEC 6429 8D, decimal 141) is used to move the printing position back one line (by reverse feeding the paper, or by moving a display cursor up one line) so that other characters may be printed over existing text. This may be done to make them bolder, or to add underlines, strike-throughs or other characters such asdiacritics.
Similarly,PLD (U+008B PARTIAL LINE FORWARD, decimal 139) andPLU (U+008C PARTIAL LINE BACKWARD, decimal 140) can be used to advance or reverse the text printing position by some fraction of the vertical line spacing (typically, half). These can be used in combination for subscripts (by advancing and then reversing) and superscripts (by reversing and then advancing), and may also be useful for printing diacritics.
[A]fter decades of frustration, and having to download a real text editor to change a single line in a config file from a Linux box, Microsoft has updated Notepad to be able to handle end of line characters used in Unix, Linux, and macOS environments.
As with any change to a long-established tool, there's a chance that this new behavior may not work for your scenarios, or you may prefer to disable this new behavior and return to Notepad's original behavior. To do this, you can change [...registry keys...] to tweak how Notepad handles pasting of text, and which EOL character to use when Enter/Return is hit
When in doubt, transfer in binary mode.