Documentation

- Made the DEVELOPER file current.
- Completed draft 1 of the EXPOSITION file.  Needs feedback.

DEVELOPER

@@ -6,7 +6,7 @@ Deprecated Code
   This is code that is going to be phased out soon, and therefore is not worth
   fixing or documenting.  Don't waste your time.
 
-  - There is no deprecated code.  There will be soon.
+  - There is no deprecated code at the moment.  I'm sure there will be soon.
 
 New Code Needs
   This is code that needs to be written, usually down at the C++ function level.
@@ -63,7 +63,49 @@ Holiday Data Needed
   ideally it could go up on holidata.net for everyone to share.
 
 Unit Tests Needed
-  There are always more unit tests needed.
+  There are always more unit tests needed.  More specifically, better unit tests
+  are always needed.  The convention is that there are four types of unit test:
+
+  1. High level tests that exercise large features, or combinations of commands.
+     For example, dependencies.t runs through a long list of commands that test
+     dependencies, but do so by using 'add', 'modify', 'done' and 'delete'.
+  2. Regression tests that ensure certain bugs are fixed and stay fixed.  These
+     tests are named bug.NNN.t where NNN refers to the Redmine issue number.
+     While it is not worth creating tests for small fixes like typos, it is for
+     changes to the logic.
+  3. Small feature tests.  When small features are added, we would like small,
+     low-level feature tests named feature.NNN.t, where NNN is the Redmine
+     issue number.
+  4. Code tests.  These are tests written in C++ that exercise C++ objects, or
+     function calls.  These are the lowest level tests.  It is important that
+     these kind of tests be extensive and thorough, because the software depends
+     on this code the most.
+
+  The tests are mainly written in Perl, and all use TAP.  Here is how to get the
+  code and run the test suite:
+     git clone git://tasktools.org:task.git
+     cd task.git
+     cmake .
+     make
+     make test
+
+  Alternately, you go to the test directory and run the following:
+
+     cd task.git/test
+     make
+     make test
+
+  Either way, a TAP file 'all.log' is written that contains all test results.
+  once this file is created, you can see the failing tests using this script:
+
+    task.git/test/problems
+
+  If you make a habit of running the test suite, consider downloading the
+  vramsteg utility to show a nice colorful progress bar, from:
+
+    git://tasktools.org/vramsteg.git     
+
+  Tests needed:
 
   - The basic.t unit tests are a misnomer, and should be either removed or
     renamed.  We have long talked of 'basic functionality' that includes add,
@@ -99,13 +141,12 @@ Current Codebase Condition
 
   '2.0.0' branch:
     - Unit tests passing >99%.
-    - Basic commands working (add, list, done).
-    - Expressions E9.{h,cpp} working.
     - DOM access DOM.{h,cpp} not working well.
     - Aiming for a release in 2011, but when it is ready.
 
 ---
 
+2011-10-16 Removed obsolete entries, added test suite description.
 2011-10-16 Removed obsolete entries.
 2011-08-21 Small changes, new work listed.
 2011-08-01 Removed code already obsoleted or completed.

EXPOSITION

@@ -25,7 +25,6 @@ Command Line Parsing
   filter or set of modifications.
 
 
-
 Dispatch
   Dispatch is simple:  once the command line is parsed, the command is used to
   look up a command object, then call its execute method.
@@ -35,25 +34,167 @@ Dispatch
   implements the functionality.
 
 
-
 Command Objects
-  Every task command is implemented by a command object.
+  Every task command is implemented by a command object.  The command object
+  provides metadata, 
+
 
 Column Objects
-
+  There is a 1:1 correspondence between attributes stored in the data files and
-
+  the columns that may be reported.  These are represented by column objects,
-GC
+  which are responsible for validating input, measuring space needed according
+  to various formats, and for rendering data for reports.
 
 
 TDB2
+  The TDB2 object is a layered, transactioned I/O manager.  Its purpose is to
+  isolate code from file I/O, locking, synching, and parsing details.  It is
+  also responsible for minimizing reads, writes and parsing of data files.
+
+  All input is assumed to be UTF8.  All stored data is UTF8.
+
+
+GC
+  Garbage Collection is the process that moves tasks between the pending.data
+  and completed.data files.  It is also responsible for waking tasks out of the
+  wait state.
+
+  Every command that displays task IDs will cause a GC to be run first, which
+  minimizes the number of changes necessary to the task IDs.  This means that
+  when a report shows task IDs, those IDs will remain valid while subsequent
+  write commands are issued.  The next report run may show different IDs.
+
+  Minimizing the size of pending.data is important for performance, because it
+  is the file that is accessed most.
+
+
+Files
+  The data files used are all kept in the rc.data.location directory, which
+  defaults to ~/.task.  The files are:
+
+    pending.data
+    completed.data
+    undo.data
+    backlog.data
+    synch.key
+
+  The pending.data file aspires to contain only pending, waiting and recurring
+  tasks, but this is only correct after a GC, and before any tasks are modified.
+  This file tends to be relatively stable in size, reflecting the length of the
+  task list.
+
+  The completed.data file accumulates data over time, and grows unbounded.
+
+  The undo.data file accumulates changes over time, and grows unbounded.  It
+  provides all the necessary metadata to support the 'undo' command and the
+  'merge' command.
+
+  The backlog.data file contains an accumulated set of changes that have not
+  been transmitted to the task server.  It grows unbounded between 'synch'
+  commands.
+
+  The synch.key file contains a synch receipt that is used to optimize synch
+  operations.
 
 
 Filter
+  A filter is simply a set of command line arguments, but is only a subset of
+  the complete command line.  These arguments (Arg objects) are grouped into
+  a set by the A3 (Args) object according to whether the command found is a
+  read or write command.
+
+  There is a Command::filter method for applying a filter to a set of tasks,
+  yielding a result set.  It does this by creating an expression from the
+  filter using the E9 object, then evaluating the expression for each task,
+  such that the result set contains only tasks for which the expression
+  evaluates to Boolean true.
 
 
 Sorting
+  Sorting is performed on a set of tasks.  More specifically, the list that is
+  sorted is a set of numeric indexes to tasks that are stored in a separate
+  list.  This minimizes the amount of data copying involved to just integers
+  rather than Task objects, but at the expense of one level of indirection.
+  Memory fragmentation is a bigger problem than the performance of vector
+  indexing.
+
+  The actual sorting is performed by std::stable_sort, but the compare function
+  is custom.
 
 
 Render
+  There are two rendering objects, ViewTask and ViewText.  These both have the
+  same tabular grid rendering capabilities.  ViewText maintains a 2D vector of
+  strings to contain the data to be rendered, so it is used for things like the
+  help command output.  ViewTask does not copy data, but assumes all data is
+  stored externally in a vector of Tasks, which minimizes data copying.
+
+  ViewTask contains projection data in the form of a set of Column objects that
+  represent the X axis.  The Y axis is represented by a vector of tasks.
+
+  The rendering process is complex.  It involves dynamically setting column
+  widths based on (1) available terminal width, (2) the columns to be included
+  in the output, (3) ability to wrap text for certain columns and (4) the size
+  of the data to be rendered, which involves added complexity when UTF8 is used.
+
+  The Column objects determine minimum width for a column and the maximum width
+  which then allows ViewT* to make choices.
 
 
+Test Suite
+  A strong and diverse test suite is critical to the successful release of any
+  software.  With the complex command set and its myriad permutations, a test
+  suite is the only way to ensure quality levels, and guarantee that big changes
+  are sound.
+
+  It is intended that the test suite continues growing, mostly adding more
+  regression tests (bug.*.t) and more feature tests (feature.*.t).  The test are
+  mostly written in Perl, and utilize the Test::More module to generate TAP
+  output.  Some tests are written in C++ and also generate TAP.
+
+  There are currently around 5,000 unit tests, that take a minute or two to run
+  in total.
+
+  There is a tinderbox that runs on a variable frequency.  As a release
+  approaches, the frequency is boosted so there are always current results to be
+  found.  Between releases the tinderbox runs daily.  It is intended that this
+  be modified for continuous integration, so it runs once per commit.
+
+    http://tasktools.org/tinderbox/task.html
+
+  When making code changes, it is important that the test suite be run to verify
+  that functionality was not broken.
+
+
+Debugging
+  The 'rc.debug=on' override provides the following additional information which
+  is useful during debugging:
+
+    - Timing of various components (used to generate the data for the charts at
+      http://tasktools.org/performance).
+    - Data load times.
+    - Terminal size, color capabilities.
+    - Command line parsing steps, shown in colorful diagrams.
+    - TDB2 layer and I/O information.
+
+
+Patches
+  Patches are encouraged and welcomed.  Either attach them to the appropriate
+  Redmine issue, or send them to support@taskwarrior.org.  A good patch:
+
+    - Maintains the MIT license, and does not contain code lifted from other
+      sources.
+    - Precisely addresses one issue only.
+    - Doesn't break unit tests.
+    - Doesn't introduce dependencies.
+    - Is accompanied by unit tests, where appropriate.
+    - Is accompanied by documentation changes, where appropriate.
+    - Conforms to the prevailing coding standards - in other words, it should
+      fit right in with the existing code.
+
+  A patch may be rejected for any of the above reasons, and more.  Bad patches
+  may be accepted and modified depending on work load and mood.  It is possible
+  that a patch may be rejected because it conflicts in some way with plans or
+  upcoming changes.
+
+---