Documentation

- Updated developer documentation for 2.4.0.

DEVELOPER

@@ -4,7 +4,7 @@ How to Build
     $ git clone https://git.tasktools.org/scm/tm/task.git task.git
     $ cd task.git
     $ git checkout 2.4.0                 # Dev branch
-    $ cmake -DCMAKE_BUILD_TYPE=debug .   # debug or release.  Default: neither.
+    $ cmake -DCMAKE_BUILD_TYPE=debug .   # debug or release. Default: neither.
     $ make VERBOSE=1                     # Shows details
 
   Running Test Suite:
@@ -14,67 +14,72 @@ How to Build
                                          # Install vramsteg for blinkenlights
     $ ./problems                         # Find errors in all.log
 
+  Note that any development should be performed using a git clone, and the
+  current development branch. The source tarballs do not reflect HEAD, and do
+  not contain the test suite.
+
 General Statement
   This file is intended to convey the current efforts, priorities and needs of
-  the codebase.  It is for anyone looking for a way to start contributing.
+  the codebase. It is for anyone looking for a way to start contributing.
   While this is biased towards developers, anyone can contribute, and everyone
-  is encouraged to do so.  Here are many ways to contribute that may not be
+  is encouraged to do so. Here are many ways to contribute that may not be
   immediately obvious to you:
 
-  - Use Taskwarrior, become familiar with it, and make suggestions.  There are
+  - Use Taskwarrior, become familiar with it, and make suggestions. There are
     always ongoing discussions about new features and changes to existing
-    features.  Join us in the forums.
+    features.
 
-  - Join us in the #taskwarrior IRC channel on freenode.net.  Some great ideas,
+  - Join us in the #taskwarrior IRC channel on freenode.net. Some great ideas,
-    suggestions, testing and discussions have taken place there.  It is also
+    suggestions, testing and discussions have taken place there. It is also
     the quickest way to get help, or confirm a bug.
 
   - Join https://answers.tasktools.org and help us by asking, answering and
     voting on questions and answers, directly helping those who ask, and
-    indirectly helping those who search for existing answers.
+    helping future users who search for existing answers.
 
   - Review documentation: there are man pages, online articles, tutorials and
     so on, and these may contain errors, or they may not convey ideas in the
-    best way.  Perhaps you can help improve it.  Contact us - documentation is
+    best way. Perhaps you can help improve it. Contact us - documentation is
     a separate effort from the codebase, and includes all web sites.
 
-  - Take a look at the bug database, and help triage the bug list.  This is a
+  - Take a look at the bug database, and help triage the bug list. This is a
     review process that involves confirming bugs, providing additional data,
-    information or analysis.  Bug triage is very useful and much needed.
+    information or analysis. Bug triage is very useful and much needed.
 
-  - Fix a bug.  For this you'll need C++ and Git skills, but this is one of
+  - Fix a bug. For this you'll need C++ and Git skills, but this is one of
-    the largest ways you can contribute.  We welcome all bug fixes, provided
+    the largest ways you can contribute. We welcome all bug fixes, provided
     the work is done well and doesn't create other problems or introduce new
-    dependencies.  We recommend talking to us before embarking on this quest.
+    dependencies. We recommend talking to us before starting.
 
-  - Add unit tests.  Unit tests are possibly the most useful contributions of
+  - Add unit tests. Unit tests are possibly the most useful contributions of
     all, because they not only improve the quality of the code, but prevent
     future regressions, therefore maintaining quality of subsequent releases.
     Plus, broken tests are a great motivator for us to fix the causal bug.
+    You'll need Python skills, as we are migrating from Perl to Python for our
+    test suite.
 
-  - Add a feature.  Well, let's be very clear about this: adding a feature is
+  - Add a feature. Well, let's be very clear about this: adding a feature is
     not usually well-received, and if you add a feature and send a patch, it
-    will most likely be rejected.  The reason for this is that there are many
+    will most likely be rejected. The reason for this is that there are many
-    efforts under way, in various code branches.  There is a very good chance
+    efforts under way, in various code branches. There is a very good chance
     that the feature you add is either already in progress, or being done in a
-    way that is more fitting when considering other work in progress.  So if
+    way that is more fitting when considering other work in progress. So if
-    you want to add a feature, please don't.  Start by talking to us, and find
+    you want to add a feature, please don't. Start by talking to us, and find
-    out what is currently under way or planned.  You might find that we've
+    out what is currently under way or planned. You might find that we've
-    already rejected such a feature for some very good reasons.  So please
+    already rejected such a feature for some very good reasons. So please
     check first, so we don't duplicate effort or waste anyone's time.
 
   - Donate some server time to the testing effort, by participating in the
-    continuous integration of all changes, with our Flod software.  See this
+    continuous integration of all changes, with our Flod software. See this
-    page for an example:  http://tasktools.org/tinderbox
+    page for an example:  http://central.tasktools.org. Ask us about running a
-    Ask us about running a Flod satellite, we're always looking for exotic
+    Flod satellite, we're always looking for exotic platforms to test on.
-    platforms to test on.
 
-  - Spread the word.  Help others become more effective at managing tasks.
+  - Spread the word. Help others become more effective at managing tasks.
 
-  - Encouragement.  Tell us what works for you, and what doesn't.  It's all
+  - Encouragement. Tell us what works for you, and what doesn't. Tell us about
-    good.
+    your methodology for managing tasks. It's all good.
 
-  - Request a feature.  This not only tells us that you think something is
+  - Request a feature. This not only tells us that you think something is
     missing from the software, but gives us insights into how you use it.
     Plus, you might get your feature implemented.
 
@@ -82,11 +87,11 @@ General Statement
 
 Deprecated Code
   This is code that is going to be phased out soon, and therefore is not worth
-  fixing or documenting.  Don't waste your time.
+  fixing or documenting. Don't waste your time.
 
-  - Shadow file support in core.  It will migrate to become an external script.
+  - Shadow file support in core. It will migrate to become an external script.
 
-  - Priorities in core.  This will be migrated to become a UDA as soon as we
+  - Priorities in core. This will be migrated to become a UDA as soon as we
     have the right support in place for custom sorting.
 
   - Attribute modifiers.
@@ -95,14 +100,14 @@ New Code Needs
   This is code that needs to be written, usually down at the C++ function/method
   level.
 
-  - Need export_viz.yy script.  Any language.  This would have value as an
+  - Need export_viz.yy script. Any language. This would have value as an
     example, or template script serving as a starting-point for anyone who
     needed this format.
-  - Need new export_xxx.yy scripts - the more the better.  Any language.
+  - Need new export_xxx.yy scripts - the more the better. Any language.
   - The JSON.{h,cpp} code is fast and efficient up to a point, but has a non-
-    linear performance curve, implying a slowdown due to fragmentation.  Find
+    linear performance curve, implying a slowdown due to fragmentation. Find
     it, fix it.
-  - Need a function that can optimize color codes.  For example, if a string
+  - Need a function that can optimize color codes. For example, if a string
     contains this:
       <red>one two </red><red>three four</red>
     It can be shortened to:
@@ -110,8 +115,8 @@ New Code Needs
     This is only possible if there is nothing between </red> and <red> and the
     colors match.
   - Need an external script that can locate and correct duplicate UUIDs in the
-    data file, as found by 'task diag'.  This should check to see if there is
+    data file, as found by 'task diag'. This should check to see if there is
-    a suitable UUID generator installed.  This should also be careful to
+    a suitable UUID generator installed. This should also be careful to
     properly handle recurring tasks.
   - Take a look at:
 
@@ -122,25 +127,25 @@ New Code Needs
     by anyone who wishes to participate.
 
 Unit Tests Needed
-  There are always more unit tests needed.  More specifically, better unit tests
+  There are always more unit tests needed. More specifically, better unit tests
-  are always needed.  The convention is that there are four types of unit test:
+  are always needed. The convention is that there are four types of unit test:
 
   1. High level tests that exercise large features, or combinations of commands.
      For example, dependencies.t runs through a long list of commands that test
      dependencies, but do so by using 'add', 'modify', 'done' and 'delete'.
-  2. Regression tests that ensure certain bugs are fixed and stay fixed.  These
+  2. Regression tests that ensure certain bugs are fixed and stay fixed. These
      tests are named bug.NNN.t where NNN refers to the Redmine issue number.
      While it is not worth creating tests for small fixes like typos, it is for
      changes to the logic.
-  3. Small feature tests.  When small features are added, we would like small,
+  3. Small feature tests. When small features are added, we would like small,
      low-level feature tests named feature.NNN.t, where NNN is the Redmine
      issue number.
-  4. Code tests.  These are tests written in C++ that exercise C++ objects, or
+  4. Code tests. These are tests written in C++ that exercise C++ objects, or
-     function calls.  These are the lowest level tests.  It is important that
+     function calls. These are the lowest level tests. It is important that
      these kind of tests be extensive and thorough, because the software depends
      on this code the most.
 
-  The tests are mainly written in Perl, and all use TAP.  We want them all to be
+  The tests are mainly written in Perl, and all use TAP. We want them all to be
   eventually replaced by Python, so we are now only accepting new tests that use
   the Python-based approach you can see in some of the existing tests. Here is
   how to get the code and run the test suite:
@@ -174,17 +179,14 @@ Unit Tests Needed
     many issues, open and closed, have the "needsTest" label. These are things
     that we would like to see in the test suite, as regression tests.
   - The basic.t unit tests are a misnomer, and should be either removed or
-    renamed.  We have long talked of 'basic functionality' that includes add,
+    renamed. We have long talked of 'basic functionality' that includes add,
-    delete, done, and list commands.  We need unit tests that prove that basic
+    delete, done, and list commands. We need unit tests that prove that basic
     functionality is working, and the file containing them should be called
     basic.t.
   - Test propagation of modifications to recurring tasks.
   - Test regex support.
-  - Need unit tests for each bug in the issue list, if suitable.
+  - Need unit tests for each bug in the issue list that is marked with the
-  - We have bugs in the unit tests that only manifest on days such as December
+    'needsTest' label.
-    31st.  Clearly there is some bad date math going on, most likely in the
-    tests themselves, rather than in Taskwarrior, although with broken tests,
-    who knows?
 
   * Note that running the unit tests requires the Perl JSON module to be
     installed.
@@ -196,8 +198,6 @@ Work in Progress
   Things that are currently in flux, which is another way of saying leave it
   alone while it is being worked on.
 
-  - Command line parser (A3).
-  - Expression evaluation (E9).
   - All columns/Col*::validate methods.
   - New columns/Col*::modify methods.
 
@@ -207,14 +207,11 @@ Current Codebase Condition
     - 2.3.0 Current release, locked.
 
   '2.3.1' branch:
-    - Bug fix branch.  To be released soon.
+    - Bug fix branch. To be released soon.
 
   '2.4.0' branch:
     - Current development branch with new command line parser, expressions
 
 ---
 
-2014-01-19 Updated for 2.4.0.
+2014-07-04 Updated for 2.4.0
-2014-04-13 Added answers.tasktools.org, corrected URLs.
-2014-05-11 Added build info.
-2014-05-31 Updated unit tests section.

EXPOSITION

@@ -3,26 +3,26 @@ Startup
   Context::initialize and Context::run methods.
 
   Context is a large object that holds all task information, both in terms of
-  the task data, and intermediate run-time data.  Having one global Context
+  the task data, and intermediate run-time data. Having one global Context
-  object means we don't have 50 global variables.  Context is therefore just a
+  object means we don't have 50 global variables. Context is therefore just a
   big global bucket of data.
 
-  Context::initialize sets up all the data and processes the command line.  The
+  Context::initialize sets up all the data and processes the command line. The
   initialization process is a big chicken-and-egg problem, because the command
   line depends on configuration (aliases) and the command line can force a
-  reload of configuration (rc:foo).  This is solved by look-ahead: the command
+  reload of configuration (rc:foo). This is solved by look-ahead: the command
   line is scanned for 'rc:xxx' and 'rc.data.location:xxx' arguments, then later
   for overrides.
 
-  The Context::run method handles all the debug output and exceptions.  Its
+  The Context::run method handles all the debug output and exceptions. Its
   main purpose is to set up exception handling and call Context::dispatch.
 
 
 Command Line Parsing
-  Command line parsing is difficult because of all the ambiguity.  The solution
+  Command line parsing is difficult because of all the ambiguity. The solution
-  is to make several passes over the command line.  For example, the task
+  is to make multiple passes over the command line. For example, the command
-  command determines whether subsequent arguments are interpreted as part of a
+  determines whether subsequent arguments are interpreted as part of a filter or
-  filter or set of modifications.
+  set of modifications.
 
 
 Dispatch
@@ -30,40 +30,40 @@ Dispatch
   look up a command object, then call its execute method.
 
   Context stores an associative map of command object pointers indexed by a
-  string.  This means the 'done' string is an index to the CmdDone object that
+  string. This means the 'done' string is an index to the CmdDone object that
   implements the functionality.
 
 
 Command Objects
-  Every task command is implemented by a command object.  The command object
+  Every task command is implemented by a command object. The command object
   provides metadata, usage and one-line help in addition to the ::execute method
   that implements the command.
 
 
 Column Objects
   There is a 1:1 correspondence between attributes stored in the data files and
-  the columns that may be reported.  These are represented by column objects,
+  the columns that may be reported. These are represented by column objects,
   which are responsible for validating input, measuring space needed according
   to various formats, and for rendering data for reports.
 
 
 TDB2
-  The TDB2 object is a layered, transactioned I/O manager.  Its purpose is to
+  The TDB2 object is a layered, transactioned I/O manager. Its purpose is to
-  isolate code from file I/O, locking, synching, and parsing details.  It is
+  isolate code from file I/O, locking, synching, and parsing details. It is
   also responsible for minimizing reads, writes and parsing of data files.
 
-  All input is assumed to be UTF8.  All stored data is UTF8.
+  All input is assumed to be UTF8. All stored data is UTF8.
 
 
 GC
   Garbage Collection is the process that moves tasks between the pending.data
-  and completed.data files.  It is also responsible for waking tasks out of the
+  and completed.data files. It is also responsible for waking tasks out of the
   wait state.
 
   Every command that displays task IDs will cause a GC to be run first, which
-  minimizes the number of changes necessary to the task IDs.  This means that
+  minimizes the number of changes necessary to the task IDs. This means that
   when a report shows task IDs, those IDs will remain valid while subsequent
-  write commands are issued.  The next report run may show different IDs.
+  write commands are issued. The next report run may show different IDs.
 
   Minimizing the size of pending.data is important for performance, because it
   is the file that is accessed most.
@@ -71,7 +71,7 @@ GC
 
 Files
   The data files used are all kept in the rc.data.location directory, which
-  defaults to ~/.task.  The files are:
+  defaults to ~/.task. The files are:
 
     pending.data
     completed.data
@@ -85,30 +85,29 @@ Files
 
   The completed.data file accumulates data over time, and grows unbounded.
 
-  The undo.data file accumulates changes over time, and grows unbounded.  It
+  The undo.data file accumulates changes over time, and grows unbounded. It
   provides all the necessary metadata to support the 'undo' command.
 
   The backlog.data file contains an accumulated set of changes that have not
-  been transmitted to Taskserver.  It grows unbounded between 'sync' commands.
+  been transmitted to Taskserver. It grows unbounded between 'sync' commands.
 
 
 Filter
   A filter is simply a set of command line arguments, but is only a subset of
-  the complete command line.  These arguments (Arg objects) are grouped into
+  the complete command line. These arguments are extracted from the parse tree
-  a set by the A3 (Args) object according to whether the command found is a
+  according to whether the command found is a read or write command.
-  read or write command.
 
-  There is a Command::filter method for applying a filter to a set of tasks,
+  There is a Filter::subset method for applying a filter to a set of tasks,
-  yielding a result set.  It does this by creating an expression from the
+  yielding a result set. It does this by creating an expression from the
-  filter using the E9 object, then evaluating the expression for each task,
+  parse tree using the Eval object, then evaluating the expression for each task,
-  such that the result set contains only tasks for which the expression
+  such that the result set contains only tasks for which the expression evaluates
-  evaluates to Boolean true.
+  to Boolean true.
 
 
 Sorting
-  Sorting is performed on a set of tasks.  More specifically, the list that is
+  Sorting is performed on a set of tasks. More specifically, the list that is
   sorted is a set of numeric indexes to tasks that are stored in a separate
-  list.  This minimizes the amount of data copying involved to just integers
+  list. This minimizes the amount of data copying involved to just integers
   rather than Task objects, but at the expense of one level of indirection.
   Memory fragmentation is a bigger problem than the performance of vector
   indexing.
@@ -118,16 +117,16 @@ Sorting
 
 
 Render
-  There are two rendering objects, ViewTask and ViewText.  These both have the
+  There are two rendering objects, ViewTask and ViewText. These both have the
-  same tabular grid rendering capabilities.  ViewText maintains a 2D vector of
+  same tabular grid rendering capabilities. ViewText maintains a 2D vector of
   strings to contain the data to be rendered, so it is used for things like the
-  help command output.  ViewTask does not copy data, but assumes all data is
+  help command output. ViewTask does not copy data, but assumes all data is
   stored externally in a vector of Tasks, which minimizes data copying.
 
   ViewTask contains projection data in the form of a set of Column objects that
-  represent the X axis.  The Y axis is represented by a vector of tasks.
+  represent the X axis. The Y axis is represented by a vector of tasks.
 
-  The rendering process is complex.  It involves dynamically setting column
+  The rendering process is complex. It involves dynamically setting column
   widths based on (1) available terminal width, (2) the columns to be included
   in the output, (3) ability to wrap text for certain columns and (4) the size
   of the data to be rendered, which involves added complexity when UTF8 is used.
@@ -138,23 +137,24 @@ Render
 
 Test Suite
   A strong and diverse test suite is critical to the successful release of any
-  software.  With the complex command set and its myriad permutations, a test
+  software. With the complex command set and its myriad permutations, a test
   suite is the only way to ensure quality levels, and guarantee that big changes
   are sound.
 
   It is intended that the test suite continues growing, mostly adding more
-  regression tests (bug.*.t) and more feature tests (feature.*.t).  The test are
+  regression and feature tests. The test are mostly written in Perl, and utilize
-  mostly written in Perl, and utilize the Test::More module to generate TAP
+  the Test::More module to generate TAP output. This is changing though, and the
-  output.  Some tests are written in C++ and also generate TAP.
+  suite is slowly migrating to Python. Some tests are written in C++ and all
+  tets generate TAP output.
 
-  There are currently over 5,000 unit tests, that take a minute or two to run
+  There are currently about 9,000 unit tests, that take a minute or two to run
   in total.
 
-  Taskwarrior uses flod software to automate continuous integration across
+  Taskwarrior uses flod software to automate continuous integration across many
-  many platforms. Code changes are automatically detected, propagated, built and
+  platforms. Code changes are automatically detected, propagated, built and
   tested on a variety of participating platforms. Grid testing results are here:
 
-    http://tasktools.org/tinderbox/taskwarrior-2.4.0.html
+    http://central.tasktools.org/task-2.4.0.html
 
   When making code changes, it is important that the test suite be run to verify
   that functionality was not broken.
@@ -168,26 +168,26 @@ Debugging
       http://tasktools.org/performance).
     - Data load times.
     - Terminal size, color capabilities.
-    - Command line parsing steps, shown in colorful diagrams.
+    - Colorful, confusing, command line parse tree.
     - TDB2 layer and I/O information.
 
 
 Patches
-  Patches are encouraged and welcomed.  Either attach them to the appropriate
+  Patches are encouraged and welcomed. Either attach them to the appropriate
-  Redmine issue, or send them to support@taskwarrior.org.  A good patch:
+  Redmine issue, or send them to support@taskwarrior.org. A good patch:
 
     - Maintains the MIT license, and does not contain code lifted from other
       sources.
     - Precisely addresses one issue only.
     - Doesn't break unit tests.
     - Doesn't introduce dependencies.
-    - Is accompanied by unit tests, where appropriate.
+    - Is accompanied by unit tests, where appropriate, written in Python.
     - Is accompanied by documentation changes, where appropriate.
     - Conforms to the prevailing coding standards - in other words, it should
       fit right in with the existing code.
 
   A patch may be rejected for not following the above guidelines, and more.
-  Bad patches may be accepted and modified depending on work load and mood.  It
+  Bad patches may be accepted and modified depending on work load and mood. It
   is possible that a patch may be rejected because it conflicts in some way with
   plans or upcoming changes.