Check point of work to date.

[dwarf-doc.git] / dwarf5 / latexdoc / introduction.tex

\chapter{Introduction}
\label{chap:introduction}
\pagenumbering{arabic}
This document defines a format for describing programs to
facilitate user source level debugging. This description
can be generated by compilers, assemblers and linkage
editors. It can be used by debuggers and other tools. 
The debugging information format does not favor the design of any
compiler or debugger. 
Instead, the goal is to create a method
of communicating an accurate picture of the source program
to any debugger in a form that is extensible to different
languages while retaining compatibility.
  
The design of the
debugging information format is open-ended, allowing for
the addition of new debugging information to accommodate new
languages or debugger capabilities while remaining compatible
with other languages or different debuggers.

\section{Purpose and Scope}
The debugging information format described in this document is
designed to meet the symbolic, source-level debugging needs of
different languages in a unified fashion by requiring language
independent debugging information whenever possible.  
Aspects
of individual languages, such as \addtoindex{C++} virtual functions or
\addtoindex{Fortran} common 
\nolink{blocks}, are accommodated by creating attributes
that are used only for those languages. 
This document is
believed to cover most debugging information needs of 
\addtoindex{Ada},
\addtoindex{C}, \addtoindex{C++}, \addtoindex{COBOL}, 
and \addtoindex{Fortran}; it also covers the basic needs
of various other languages.

This document describes \DWARFVersionV,
the fifth generation
of debugging information based on the DWARF format. 
\DWARFVersionV{} extends \DWARFVersionIV{}
in a compatible manner.

The intended audience for this document is the developers
of both producers and consumers of debugging information,
typically compilers, debuggers and other tools that need to
interpret a binary program in terms of its original source.


\section{Overview}

There are two major pieces to the description of the DWARF
format in this document. The first piece is the informational
content of the debugging entries. The second piece is the
way the debugging information is encoded and represented in
an object file.

The informational content is described in Chapters
\ref{chap:generaldescription} through
\ref{chap:otherdebugginginformation}. Chapter 
\ref{chap:generaldescription}
describes the overall structure of the information
and attributes that are common to many or all of the different
debugging information entries. Chapters
\ref{chap:programscopeentries}, 
\ref{chap:dataobjectandobjectlistentries} and 
\ref{chap:typeentries} describe
the specific debugging information entries and how they
communicate the necessary information about the source program
to a debugger. Chapter \ref{chap:otherdebugginginformation} 
describes debugging information
contained outside of the debugging information entries. The
encoding of the DWARF information is presented in Chapter
\ref{datarep:datarepresentation}.

This organization closely follows that used in the 
\DWARFVersionIV{} document. Except where needed to incorporate
new material or to correct errors, the \DWARFVersionIV{}
text is generally reused in this document with little or
no modification.

In the following sections, text in normal font describes
required aspects of the DWARF format.  Text in \textit{italics} is
explanatory or supplementary material, and not part of the
format definition itself. The several appendices consist only
of explanatory or supplementary material, and are not part
of the formal definition.

\section{Objectives and Rationale}

DWARF has had a set of objectives since its inception which have 
guided the design and evolution of the debugging format.  A discussion 
of these objectives and the rationale behind them may help with an 
understanding of the DWARF Debugging Format. 

Although DWARF Version 1 was developed in the late 1980's as a 
format to support debugging C programs written for AT\&T hardware 
running SVR4, \DWARFVersionII{} and later has evolved far beyond 
this origin. One difference between DWARF and other formats 
is that the latter are often specific to a particular language, 
architecture, and/or operating system. 

\subsection{Language Independence}
DWARF is applicable to a broad range of existing procedural 
languages and is designed to be extensible to future languages.  
These languages may be considered to be "C-like" but the 
characteristics of C are not incorporated into DWARF Version 2 
and later, unlike DWARF Version 1 and other debugging formats.  
DWARF abstracts concepts as much as possible so that the 
description can be used to describe a program in any language. 
As an example, the DWARF descriptions used to describe C functions, 
Pascal subroutines, and Fortran subprograms are all the same, 
with different attributes used to specify the differences between 
these similar programming language features. 

On occasion, there is a feature which is specific to one 
particular language and which doesn't appear to have more 
general application.  For these, DWARF has a description 
designed to meet the language requirements, although, to the 
extent possible, an effort is made to generalize the attribute. 
An example of this is the \DWTAGconditionNAME{} 
debugging information entry, 
used to describe \addtoindex{COBOL} level 88 conditions, which 
is described in abstract terms rather than COBOL-specific terms.  
Conceivably, this TAG might be used with a different language 
which had similar functionality. 

\subsection{Architecture Independence}
DWARF can be used with a wide range of processor architectures, 
whether byte or word oriented, linear or segmented, with any 
word or byte size.  DWARF can be used with Von Neumann architectures, 
using a single address space for both code and data; Harvard 
architectures, with separate code and data address spaces; and 
potentially for other architectures such as DSPs with their 
idiosyncratic memory organizations.  DWARF can be used with 
common register-oriented architectures or with stack architectures. 

DWARF assumes that memory has individual units (words or bytes) 
which have unique addresses which are ordered.  (Some architectures 
like the i386 can represent the same physical machine location with 
different segment and offset pairs.  Identifying aliases is an 
implementation issue.)

\needlines{6}
\subsection{Operating System Independence}
DWARF is widely associated with SVR4 Unix and similar operating 
systems like BSD and Linux.  DWARF fits well with the section 
organization of the ELF object file format. Nonetheless, DWARF 
attempts to be independent of either the OS or the object file 
format.  There have been implementations of DWARF debugging 
data in COFF, Mach-O and other object file formats. 

DWARF assumes that any object file format will be able to 
distinguish the various DWARF data sections in some fashion, 
preferably by name. 

DWARF makes a few assumptions about functionality provided by 
the underlying operating system.  DWARF data sections can be 
read sequentially and independently.
Each DWARF data section is a sequence of 8-bit bytes, 
numbered starting with zero.  The presence of offsets from one 
DWARF data section into other data sections does not imply that 
the underlying OS must be able to position files randomly; a 
data section could be read sequentially and indexed using the offset. 

\subsection{Compact Data Representation} 
The DWARF description is designed to be a compact file-oriented 
representation. 

There are several encodings which achieve this goal, such as the 
TAG and attribute abbreviations or the line number encoding.  
References from one section to another, especially to refer to 
strings, allow these sections to be compacted to eliminate 
duplicate data. 

There are multiple schemes for eliminating duplicate data or 
reducing the size of the DWARF debug data associated with a 
given file.  These include COMDAT, used to eliminate duplicate 
function or data definitions, the split DWARF object files 
which allow a consumer to find DWARF data in files other than 
the executable, or the type units, which allow similar type 
definitions from multiple compilations to be combined. 

In most cases, it is anticipated that DWARF 
debug data will be read by a consumer (usually a debugger) and 
converted into a more efficiently accessed internal representation.  
For the most part, the DWARF data in a section is not the same as 
this internal representation.

\needlines{4}
\subsection{Efficient Processing} 
DWARF is designed to be processed efficiently, so that a 
producer (a compiler) can generate the debug descriptions 
incrementally and a consumer can read only the descriptions 
which it needs at a given time. The data formats are designed 
to be efficiently interpreted by a consumer.

As mentioned, there is a tension between this objective and 
the preceding one.  A DWARF data representation which resembles 
an internal data representation may lead to faster processing, 
but at the expense of larger data files. This may also constrain 
the possible implementations.

\subsection{Implementation Independence}
DWARF attempts to allow developers the greatest flexibility 
in designing implementations, without mandating any particular 
design decisions. Issues which can be described as 
quality-of-implementation are avoided.

\subsection{Explicit Rather Than Implicit Description}
DWARF describes the source to object translation explicitly 
rather than using common practice or convention as an implicit 
understanding between producer and consumer.  For example, where 
other debugging formats assume that a debugger knows how to 
virtually unwind the stack, moving from one stack frame to the next using 
implicit knowledge about the architecture or operating system, 
DWARF makes this explicit in the Call Frame Information description.

\subsection{Avoid Duplication of Information}
DWARF has a goal of describing characteristics of a program once, 
rather than repeating the same information multiple times.  The 
string sections can be compacted to eliminate duplicate strings, 
for example.  Other compaction schemes or references between 
sections support this. Whether a particular implementation is 
effective at eliminating duplicate data, or even attempts to, 
is a quality-of-implementation issue.  

\subsection{Leverage Other Standards}
Where another standard exists which describes how to interpret 
aspects of a program, DWARF defers to that standard rather than 
attempting to duplicate the description.  For example, C++ has 
specific rules for deciding which function to call depending 
name, scope, argument types, and other factors.  DWARF describes 
the functions and arguments, but doesn't attempt to describe 
how one would be selected by a consumer performing any particular 
operation.

\subsection{Limited Dependence on Tools}
DWARF data is designed so that it can be processed by commonly 
available assemblers, linkers, and other support programs, 
without requiring additional functionality specifically to 
support DWARF data.  This may require the implementer to be 
careful that they do not generate DWARF data which cannot be 
processed by these programs.  Conversely, an assembler which 
can generate LEB128 (Little-Endian Base 128) 
values may allow the compiler to generate 
more compact descriptions, and a linker which understands the 
format of string sections can merge these sections.  Whether 
or not an implementation includes these functions is a 
quality-of-implementation issue, not mandated by the DWARF 
specification.

\subsection{Separate Description From Implementation}
DWARF intends to describe the translation of a program from 
source to object, while neither mandating any particular design 
nor making any other design difficult.  For example, DWARF 
describes how the arguments and local variables in a function 
are to be described, but doesn't specify how this data is 
collected or organized by a producer.  Where a particular DWARF 
feature anticipates that it will be implemented in a certain 
fashion, informative text will suggest but not require this design.

\subsection{Permissive Rather Than Prescriptive}
The DWARF Standard specifies the meaning of DWARF descriptions. It does not
specify in detail what a particular producer 
\bb
must
\eb
generate for any source to
object conversion.  One producer may generate a more complete description
than another, it may describe features in a different order (unless the 
standard explicitly requires a particular order), or it may use
different abbreviations or compression methods.  Similarly, DWARF does not
specify exactly what a particular consumer should do with each part of the
description, although we believe that the potential uses for each description
should be evident.

DWARF is permissive, allowing different producers to generate different
descriptions for the same source to object conversion, and permitting
different consumers to provide more or less functionality or information
to the user.  This may result in debugging information being larger or
smaller, compilers or debuggers which are faster or slower, and more or
less functional.  These are described as differences in "Quality of
Implementation".

Each producer conforming to the DWARF standard must follow the format and
meaning as specified in the standard.  As long as the DWARF description
generated follows this specification, the producer is generating valid DWARF.
For example, DWARF allows a producer to identify the end of a function
prologue in the Line Information so that a debugger can stop at this location.
A producer which does this is generating valid DWARF, as is another which
doesn't.  As another example, one producer may generate descriptions
for variables which are moved from memory to a register in a certain range,
while another may only describe the variable's location in memory.  Both are
valid DWARF descriptions, while a consumer using the former would be able
to provide more accurate values for the variable while executing in that
range than a consumer using the latter.

\bb
In this document, where the word \doublequote{may} is used, the producer has 
the option to follow the description or not.  Where the text says 
\doublequote{may not}, this is prohibited.  Where the text says \doublequote{should}, 
this is advice about best practice, but is not a requirement.
\eb

\subsection{Vendor Extensibility}
This document does not attempt to cover all interesting
languages or even to cover all of the possible debugging
information needs for its primary target languages. 
Therefore,
the document provides vendors a way to define their own
debugging information tags, attributes, base type encodings,
location operations, language names, calling conventions and
call frame instructions by reserving a subset of the valid
values for these constructs for vendor specific additions
and defining related naming conventions. 
Vendors may also use
debugging information entries and attributes defined here in
new situations. 
Future versions of this document will not use
names or values reserved for vendor specific additions. 
All names and values not reserved for vendor additions, however,
are reserved for future versions of this document.

Where this specification provides a means for
describing the source language, implementors are expected
to adhere to that specification. 
For language features that
are not supported, implementors may use existing attributes
in novel ways or add vendor-defined attributes. 
Implementors
who make extensions are strongly encouraged to design them
to be compatible with this specification in the absence of
those extensions.

The DWARF format is organized so that a consumer can skip over
data which it does not recognize. This may allow a consumer
to read and process files generated according to a later
version of this standard or which contain vendor extensions,
albeit possibly in a degraded manner.

\section{Changes From Version 4 to Version 5}
\addtoindexx{DWARF Version 5}
The following is a list of the major changes made to the 
DWARF Debugging Information Format since Version 4 was published. 
The list is not meant to be exhaustive.
\begin{itemize}
\item The \dotdebugtypes{}
%\addtoindexi{\texttt{.debug\_types}}{\texttt{.debug\_types} (Version 4)}
section introduced in \DWARFVersionIV{} 
is eliminated and its contents instead contained in \dotdebuginfo{} sections.
\item Add support for collecting common DWARF information 
(debugging information entries and macro definitions)
across multiple executable and shared files and keeping it in a single
\addtoindex{supplementary object file}.
\needlines{6}
\item A new line number program header format 
provides the ability to use an MD5 hash to validate 
the source file version in use, allows pooling 
of directory and file name strings and makes provision for vendor-defined
extensions. It also adds a string section specific to the line number table 
(\dotdebuglinestr)
to properly support the common practice of stripping all DWARF sections
except for line number information.
\needlines{4}
\item Add a split object file and package representations to allow most 
DWARF information to be kept separate from an executable 
or shared image. This includes new sections 
\dotdebugaddr, \dotdebugstroffsets, \dotdebugabbrevdwo, \dotdebuginfodwo, 
\dotdebuglinedwo, \dotdebugloclistsdwo, \dotdebugmacrodwo, \dotdebugstrdwo,
\dotdebugstroffsetsdwo, \dotdebugcuindex{} and \dotdebugtuindex{} 
together with new forms of attribute value for referencing these sections.
This enhances DWARF support by reducing executable program size and
by improving link times.
\item Replace the \dotdebugmacinfo{} macro information representation with
with a \dotdebugmacro{} representation that can potentially be much more compact.

\item Replace the \dotdebugpubnames{} and \dotdebugpubtypes{} sections
with a single and more functional name index section, \dotdebugnames{}.
\bb
\item Replace the location list and range list sections (\texttt{.debug\_loc} 
and \texttt{.debug\_ranges}, respectively) with new sections (\dotdebugloclists{} 
and \dotdebugrnglists) and new representations that
save space and processing time by eliminating most related
object file relocations.
\eb
\item Add a new debugging information entry (\DWTAGcallsiteNAME), related 
attributes and DWARF expression operators to describe call site information, 
including identification of tail calls and tail recursion.
\item Add improved support for \addtoindex{FORTRAN} assumed rank arrays 
(\DWTAGgenericsubrangeNAME), dynamic rank arrays (\DWATrankNAME)
and co-arrays (\DWTAGcoarraytypeNAME{}).
\item Add new operations that allow support for 
a DWARF expression stack containing typed values.
\item Add improved support for the \addtoindex{C++}:
\texttt{auto} return type, deleted member functions (\DWATdeletedNAME), 
as well as defaulted constructors and destructors (\DWATdefaultedNAME).
\item Add a new attribute (\DWATnoreturnNAME{}), to identify 
a subprogram that does not return to its caller.
\item Add language codes for C 2011, C++ 2003, C++ 2011, C++ 2014,
Dylan, Fortran 2003, Fortran 2008, Go, Haskell, 
Julia, Modula 3, Ocaml, OpenCL, Rust and Swift.
\item Numerous other more minor additions to improve functionality
and performance.
\end{itemize}

DWARF Version 5 is compatible with DWARF Version 4 except as follows:
\begin{itemize}
\item The compilation unit header (in the \dotdebuginfo{} section) has
a new \HFNunittype{} field.
\needlines{4}
\item New operand forms for attribute values are defined 
(\DWFORMaddrxNAME, \DWFORMdatasixteenNAME, \DWFORMimplicitconstNAME, 
\DWFORMlinestrpNAME,
\bb
\DWFORMloclistxNAME, \DWFORMrnglistxNAME,
\eb 
\DWFORMrefsupNAME, \DWFORMstrpsupNAME{} and \DWFORMstrxNAME).
\eb

\textit{Because a pre-DWARF Version 5 consumer will not be able to interpret 
these even to ignore and skip over them, new forms must be 
considered incompatible additions.}
\item The line number table header is substantially revised.
\needlines{4}
\item A location list entry 
with the address range \mbox{(0, \textit{maximum-address})} is defined 
as the new default location list entry.
\item In a string type, the \DWATbytesizeNAME{} attribute is re-defined 
to always describe the size of the string type. 
(Previously it described the size of the optional string length data 
field if the \DWATstringlengthNAME{} attribute was also present.)
\end{itemize}

While not strictly an incompatibility, the macro information 
representation is completely new; further, producers 
and consumers may optionally continue to support the older 
representation. While the two representations cannot both be 
used in the same compilation unit, they can co-exist in 
executable or shared images.

Similar comments apply to replacement of the \dotdebugpubnames{} 
and \dotdebugpubtypes{} sections with the new \dotdebugnames{} 
section.

\needlines{4}
\section{Changes from Version 3 to Version 4}
\addtoindexx{DWARF Version 4}
The following is a list of the major changes made to the 
DWARF Debugging Information Format since Version 3 was 
published. The list is not meant to be exhaustive.
\begin{itemize}
\item Reformulate 
Section 2.6 (Location Descriptions) 
to better distinguish DWARF location descriptions, which
compute the location where a value is found (such as an 
address in memory or a register name) from DWARF expressions, 
which compute a final value (such as an array bound).
\item Add support for bundled instructions on machine architectures 
where instructions do not occupy a whole number of bytes.
\item Add a new attribute form for section offsets, 
\DWFORMsecoffsetNAME,\addtoindexx{section offset}
to replace the use of 
\DWFORMdatafourNAME{} and \DWFORMdataeightNAME{} for section offsets.
\item Add an attribute, \DWATmainsubprogramNAME, to identify the main subprogram of a
program.
\item Define default array lower bound values for each supported language.
\item Add a new technique using separate type units, type signatures and \COMDAT{} sections to
improve compression and duplicate elimination of DWARF information.
\item Add support for new \addtoindex{C++} language constructs, including rvalue references, generalized
constant expressions, Unicode character types and template aliases.
\item Clarify and generalize support for packed arrays and structures.
\item Add new line number table support to facilitate profile based compiler optimization.
\item Add additional support for template parameters in instantiations.
\item Add support for strongly typed enumerations in languages (such as \addtoindex{C++}) that have two
kinds of enumeration declarations.
\end{itemize}
\addtoindex{DWARF Version 4} is compatible with 
\addtoindex{DWARF Version 3} except as follows:
\begin{itemize}
\item DWARF attributes that use any of the new forms of attribute value representation (for
section offsets, flag compression, type signature references, and so on) cannot be read by
\addtoindex{DWARF Version 3}
consumers because the consumer will not know how to skip over the
unexpected form of data.
\item DWARF frame and line number table sections include additional fields that affect the location
and interpretation of other data in the section.
\end{itemize}

\section{Changes from Version 2 to Version 3}
\addtoindexx{DWARF Version 3}
The following is a list of the major differences between
Version 2 and Version 3 of the DWARF Debugging Information
Format. The list is not meant to be exhaustive.
\begin{itemize}
\item
Make provision for DWARF information files that are larger
than 4 GBytes.
\item
Allow attributes to refer to debugging information entries
in other shared libraries.
\item
Add support for \addtoindex{Fortran 90} modules as well as allocatable
array and pointer types.
\item
Add additional base types for \addtoindex{C} (as revised for 1999).
\item
Add support for \addtoindex{Java} and \addtoindex{COBOL}.
\item
Add namespace support for \addtoindex{C++}.
\item
Add an optional section for global type names (similar to
the global section for objects and functions).
\item
Adopt \addtoindex{UTF-8} as the preferred representation of program name strings.
\item
Add improved support for optimized code (discontiguous
scopes, end of prologue determination, multiple section
code generation).  
\item Improve the ability to eliminate
duplicate DWARF information during linking.  
\end{itemize}

\addtoindex{DWARF Version 3}
is compatible with 
\addtoindex{DWARF Version 2} except as follows:
\begin{itemize}
\item
Certain very large values of the initial length fields that
begin DWARF sections as well as certain structures are reserved
to act as escape codes for future extension; one such extension
is defined to increase the possible size of DWARF descriptions
(see Section \refersec{datarep:32bitand64bitdwarfformats}).
\item
References that use the attribute form 
\DWFORMrefaddrNAME{}
are specified to be four bytes in the DWARF 32-bit format and
eight bytes in the DWARF 64-bit format, while 
\addtoindex{DWARF Version 2} 
specifies that such references have the same size as an
address on the target system (see Sections 
\refersec{datarep:32bitand64bitdwarfformats} and 
\refersec{datarep:attributeencodings}).
\item
The return\_address\_register field in a Common Information
Entry record for call frame information is changed to unsigned
LEB representation (see Section 
\refersec{chap:structureofcallframeinformation}).
\end{itemize}

\section{Changes from Version 1 to Version 2}
\addtoindex{DWARF Version 2} 
describes the second generation of debugging
information based on the DWARF format. While 
\addtoindex{DWARF Version 2}
provides new debugging information not available in
Version 1, the primary focus of the changes for Version
2 is the representation of the information, rather than
the information content itself. The basic structure of
the Version 2 format remains as in Version 1: the debugging
information is represented as a series of debugging information
entries, each containing one or more attributes (name/value
pairs). The Version 2 representation, however, is much more
compact than the Version 1 representation. In some cases,
this greater density has been achieved at the expense of
additional complexity or greater difficulty in producing and
processing the DWARF information. The definers believe that the
reduction in I/O and in memory paging should more than make
up for any increase in processing time.  

\needlines{5}
The representation
of information changed from Version 1 to Version 2, so that
Version 2 DWARF information is not binary compatible with
Version 1 information. To make it easier for consumers to
support both Version 1 and Version 2 DWARF information, the
Version 2 information has been moved to a different object
file section, \dotdebuginfo{}.  

\textit{
A summary of the major changes made in 
\addtoindex{DWARF Version 2}
compared to the DWARF Version 1 may be found in the 
\addtoindex{DWARF Version 2}
document.
}