root/trunk/src/astrometry/README

Astrometry.net code release 0.2
===============================

Copyright 2006-2008 Michael Blanton, David W. Hogg, Dustin Lang, Keir
Mierle and Sam Roweis.

This code is meant as an accompanyment for an upcoming paper and its
main purpose is to allow reproduction of the results in that paper.

This is research code only and is not intended for scientific use in
reducing data or other analyses.

This release includes a snapshot of all of the components of our
current research code, including routines to:
  * Convert raw USNO-B and Tycho2 into FITS format for easier use
  * Uniformize, deduplicate, and cut the FITSified catalogs
  * Build indexes from these cuts
  * Solve the astrometry of images using these indexes

The code includes:
  * A simple but powerful HEALPIX implementation
  * The QFITS library with several modifications
  * libkd, a compact and high-performance kdtree library

In principle, the code in the tarball is sufficient to reproduce our
entire astrometry system and replicate our blind astrometry results
(e.g. on the SDSS fields).  However, this requires the entire USNO-B
1.0 catalogue which is about 80GB in size.  For ease of use, we are
distributing our pre-cooked indexes (about 25 GB total), which are the
same files we're using for our web service.

INSTALLING
----------

To compile everything and run with full functionality, you will need:
  * gcc
  * cairo
  * netpbm
  * libpng
  * libjpeg
  * python (probably >= 2.4)
  * numpy

-> On Ubuntu or Debian-like systems, the following should suffice:

    $ sudo apt-get install libcairo2-dev libnetpbm10-dev netpbm \
                           libpng12-dev libjpeg-dev python-numpy

-> On RHEL, you'll need something like:
    $ sudo yum install cairo.x86_64 cairo-devel.x86_64 netpbm.x86_64 \
                       netpbm-devel.x86_64 fontconfig-devel.x86_64 \
                       libXrender-devel.x86_64 xorg-x11-proto-devel.x86_64

If you don't have and can't get these libraries, you should still be
able to compile and use the core parts of the solver, but you will
miss out on some eye-candy.

Grab the code:
  $ wget http://astrometry.net/downloads/astrometry.net-0.2.tar.bz2
  $ tar xjf astrometry.net-0.2.tar.bz2
  $ cd astrometry.net-0.2

Build the solving system:
  $ make

If you installed the libraries listed above, build the plotting code:
  $ make extra

Install it:
  $ make install

You might see some error message during compilation; see the section
ERROR MESSAGES below for fixes to common problems.

By default it will be installed in  /usr/local/astrometry .
You can override this by either:
     * editing the top-level Makefile (look for INSTALL_DIR)
  or * defining INSTALL_DIR on the command-line:
        For bash shell:
          $ export INSTALL_DIR=/path/to/astrometry
          $ make install
        or
          $ INSTALL_DIR=/path/to/astrometry make install
        For tcsh shell:
          $ setenv INSTALL_DIR /path/to/astrometry
          $ make install

The astrometry solver is composed of several executables.  You may
want to add the INSTALL_DIR/bin directory to your path:
   For bash shell:
     $ export PATH="$PATH:/usr/local/astrometry/bin"
   For tcsh shell:
     $ setenv PATH "$PATH:/usr/local/astrometry/bin"


GETTING INDEXES
---------------
Next, grab some pre-cooked indexes.  See the file GETTING-INDEXES for
details about how to do this.

(If you have indexes from a previous version of the Astrometry.net
software, see the "Upgrading" section below.)

Each index is designed to solve images within a narrow range of
scales.  The indexes designed to solve small (angular size) images are
rather large files, so you probably only want to grab the indexes
required for the images you wish to solve.  If you grab extra indexes,
the solver will run more slowly, but the results should be the same.

Each index contains a large number of "skymarks" (landmarks for the
sky) that allow our solver to identify your images.  The skymarks
contained in each index have sizes (diameters) within a narrow range.
You probably want to download indexes whose quads are, say, 10% to
100% of the sizes of the images you want to solve.

For example, let's say you have some 1-degree square images.  You
should grab indexes that contain skymarks of size 0.1 to 1 degree, or
6 to 60 arcminutes.  Referring to the table below, you should grab
indexes 203 through 209.  You might find that the same number of
fields solve, and faster, using just one or two of the indexes in the
middle of that range - in our example you might try 205, 206 and 207.

For reference, we used index 202 alone for our SDSS tests (13x9 arcmin
fields).

---------------------------------------------------------------
Index Filename          Range of skymark diameters    File Size
                               (arcminutes)              (MB)
---------------------------------------------------------------
index-210+.tar.bz2           60  - 2000                    29
index-209.tar.bz2            42  -   60                    30
index-208.tar.bz2            30  -   42                    61
index-207.tar.bz2            22  -   30                   125
index-206.tar.bz2            16  -   22                   255
index-205.tar.bz2            11  -   16                   526
index-204.tar.bz2             8  -   11                 1,023
index-203.tar.bz2             5.6  -  8.0               2,089
index-202.tar.bz2             4.0  -  5.6               4,260
index-201.tar.bz2             2.8  -  4.0               5,878
index-200.tar.bz2             2.0  -  2.8              10,058
--------------------------------------------------------------

Download the indexes you need and put them in the top-level
(astrometry-0.2) source directory.

Install them:
   $ make install-indexes

(Or you can extract them into the INSTALL_DIR/data directory by hand.)


Next, you can (optionally) configure the solver by editing the file
   INSTALL_DIR/etc/backend.cfg


UPGRADING INDEX FILES
---------------------

The format of the index files has changed from the 0.1 and 0.11
releases.  There is a program for updating them, and there is also a
makefile target.

  $ make upgrade-indexes

If you want to do it by hand, use the program "fix-bb".  Run it on
each of the ckdt.fits and skdt.fits files in your INSTALL_DIR/data
directory.


SOLVING
-------

Finally, solve some fields.

(If you didn't build the plotting commands, add "--no-plots" to the
command lines below.)

If you have any of indexes 213 to 218:
   $ solve-field --scale-low 10 demo/apod4.jpg

If you have index 219:
   $ solve-field --scale-low 30 demo/apod5.jpg

If you have any of indexes 210 to 214:
   $ solve-field --scale-low 1 demo/apod3.jpg

If you have any of indexes 206 to 211:
   $ solve-field --scale-low 1 demo/apod2.jpg

If you have any of indexes 203 to 205:
   $ solve-field apod1.jpg

If you have any of indexes 200 to 203:
   $ solve-field demo/sdss.jpg


Copyrights and credits for the demo images are listed in the file demo/CREDITS .

Note that you can also give solve-field a URL rather than a file as input:
   $ solve-field --out apod1b http://antwrp.gsfc.nasa.gov/apod/image/0302/ngc2264_croman_c3.jpg 


If you don't have the netpbm tools (eg jpegtopnm), do this instead:

If you have any of indexes 213 to 218:
   $ solve-field --scale-low 10 demo/apod4.xyls

If you have index 219:
   $ solve-field --scale-low 30 demo/apod5.xyls

If you have any of indexes 210 to 214:
   $ solve-field --scale-low 1 demo/apod3.xyls

If you have any of indexes 206 to 211:
   $ solve-field --scale-low 1 demo/apod2.xyls

If you have any of indexes 203 to 205:
   $ solve-field demo/apod1.xyls

If you have any of indexes 200 to 203:
   $ solve-field demo/sdss.xyls


OUTPUT FILES
------------

   <base>-ngc.png   : an annotation of the image.
   <base>.wcs       : a FITS WCS header for the solution.
   <base>.new       : a new FITS file containing the WCS header.
   <base>-objs.png  : a plot of the sources (stars) we extracted from
                      the image.
   <base>-indx.png  : sources (red), plus stars from the index (green),
                      plus the skymark ("quad") used to solve the
                      image.
   <base>-indx.xyls : a FITS BINTABLE with the pixel locations of
                      stars from the index.
   <base>.rdls      : a FITS BINTABLE with the RA,Dec of sources we
                      extracted from the image.
   <base>.axy       : a FITS BINTABLE of the sources we extracted, plus
                      headers that describe the job (how the image is
                      going to be solved).
   <base>.solved    : exists and contains (binary) 1 if the field solved.
   <base>.match     : a FITS BINTABLE describing the quad match that
                      solved the image.
   <base>.kmz       : (optional) KMZ file for Google Sky-in-Earth.  You need
                      to have "wcs2kml" in your PATH.  See
                       http://code.google.com/p/wcs2kml/downloads/list
                       http://code.google.com/p/google-gflags/downloads/list


TRICKS AND TIPS
---------------

*** To lower the CPU time limit before giving up:

    $  solve-field --cpu-limit 30 ...

will make it give up after 30 seconds.

(Note, however, that the "backend" configuration file (backend.cfg)
puts a limit on the CPU time that is spent on an image; solve-field
can reduce this but not increase it.)


*** Scale of the image: if you provide bounds (lower and upper limits)
on the size of the image you are trying to solve, solving can be much
faster.  In the last examples above, for example, we specified that
the field is at least 30 degrees wide: this means that we don't need
to search for matches in the indexes that contain only tiny skymarks.

Eg, to specify that the image is between 1 and 2 degrees wide:

   $ solve-field --scale-units degwidth --scale-low 1 --scale-high 2 ...

If you know the pixel scale instead:

   $ solve-field --scale-units arcsecperpix \
       --scale-low 0.386 --scale-high 0.406 ...

When you tell solve-field the scale of your image, it uses this to
decide which indexes to try to use to solve your image; each index
contains quads whose scale is within a certain range, so if these
quads are too big or too small to be in your image, there is no need
to look in that index.  It is also used while matching quads: a small
quad in your image is not allowed to match a large quad in the index
if such a match would cause the image scale to be outside the bounds
you specified.  However, all these checks are done before computing a
best-fit WCS solution and polynomial distortion terms, so it is
possible (though rare) for the final solution to fall outside the
limits you specified.  This should only happen when the solution is
correct, but you gave incorrect inputs, so you shouldn't be
complaining! :)


*** Guess the scale: solve-field can try to guess your image's scale
from a number of different FITS header values.  When it's right, this
often speeds up solving a lot, and when it's wrong it doesn't cost
much.  Enable this with:

   $ solve-field --guess-scale ...

*** Depth.  The solver works by looking at sources in your image,
starting with the brightest.  It searches for all "skymarks" that can
be built from the N brightest stars before considering star N+1.  When
using several indexes, it can be much faster to search for many
skymarks in one index before switching to the next one.  This flag
lets you control when the solver switches between indexes.  It also
lets you control how much effort the solver puts in before giving up -
by default it looks at all the sources in your image, and usually
times out before this finishes.

Eg, to first look at sources 1-20 in all indexes, then sources 21-30
in all indexes, then 31-40:

    $ solve-field --depth 20,30,40 ...
 or $ solve-field --depth 1-20 --depth 21-30 --depth 31-40 ...

Sources are numbered starting at one, and ranges are inclusive.  If
you don't give a lower limit, it will take 1 + the previous upper
limit.  To look at a single source, do

    $ solve-field --depth 42-42 ...


*** Our source extractor sometimes estimates the background badly, so
by default we sort the stars by brightness using a compromise between
the raw and background-subtracted flux estimates.  For images without
much nebulosity, you might find that using the background-subtracted
fluxes yields faster results.  Enable this by:

    $ solve-field --resort ...


*** If you've got big images: you might want to downsample them before
doing source extraction:

   $ solve-field --downsample 2 ...
or $ solve-field --downsample 4 ...
or etc


*** When solve-field processes FITS files, it runs them through a
"sanitizer" which tries to clean up non-standards-compliant images.
If your FITS files are compliant, this is a waste of time, and you can
avoid doing it.

   $ solve-field --no-fits2fits ...


*** When solve-field processes FITS images, it looks for an existing
WCS header.  If one is found, it tries to verify that header before
trying to solve the image blindly.  You can prevent this with:

   $ solve-field --no-verify ...

Note that currently solve-field only understands a small subset of
valid WCS headers: essentially just the TAN projection with a CD
matrix (not CROT).


*** If you don't want the plots to be produced:

   $ solve-field --no-plots ...


*** "I know where my image is to within 1 arcminute, how can I tell
solve-field to only look there?"

In this release of the software, you can't.  We've got plans for v0.3,
but don't hold your breath!  There is one thing you can do: if you
know that your image is within one HEALPix
(http://healpix.jpl.nasa.gov/), and if you're using the
small-angular-scale indexes with filenames like index-###-##.*.fits,
then each of these indexes covers one HEALPix (the second number is
the HEALPix number); you can edit the "backend.cfg" config file to
tell the "backend" program to only load the index that contains your
image.


*** To convert a list of pixel coordinates to RA,Dec coordinates:

   $ wcs-xy2rd -w wcs-file -i xy-list -o radec-list

Where xy-list is a FITS BINTABLE of the pixel locations of sources;
recall that FITS specifies that the center of the first pixel is pixel
coordinate (1,1).


*** To convert from RA,Dec to pixels:

   $ wcs-rd2xy -w wcs-file -i radec-list -o xy-list


*** To make cool overlay plots: see plotxy, plot-constellations.


*** To change the output filenames when processing multiple input
files: each of the output filename options listed below can include
"%s", which will be replaced by the base output filename.  (Eg, the
default for --wcs is "%s.wcs").  If you really want a "%" character in
your output filename, you have to put "%%".

Outputs include: --new-fits, --kmz, --solved, --cancel, --match,
--rdls, --corr, --wcs --keep-xylist --pnm
Also included: --solved-in, --verify


*** Reusing files between runs:

The first time you run solve-field, save the source extraction
results:
   $ solve-field --keep-xylist %s.xy input.fits ...

On subsequent runs, instead of using the original input file, use the
saved xylist instead.  Also add --continue to overwrite any output
file that already exists.
   $ solve-field input.xy --no-fits2fits --continue ...

To skip previously solved inputs (note that this assumes single-HDU
inputs):
   $ solve-field --skip-solved ...


"I wanna go fast!"
------------------

Sadly, gcc doesn't have a flag --super-fast to generate the fastest
possible code for the machine on which it is running.  Also, we are
shipping the code with assertion and debugging turned on -- this helps
ensure that bugs are caught and reported, but it does slow down the
code.  Here are some things you can do to make the code run faster:

  * in util/makefile.common, set RICKY_BOBBY to yes.

  * we try to guess "-mtune" settings that will work for you; if we're
    wrong, you can set the environment variable ARCH_FLAGS before
    compiling:

      $ ARCH_FLAGS="-mtune=nocona" make

    You can find details in the gcc manual:
      http://gcc.gnu.org/onlinedocs/

    You probably want to look in the section:
      "GCC Command Options"
         -> "Hardware Models and Configurations"
             -> "Intel 386 and AMD x86-64 Options"

    http://gcc.gnu.org/onlinedocs/gcc-4.3.0/gcc/i386-and-x86_002d64-Options.html#i386-and-x86_002d64-Options

XYLISTS
-------

The solve-field program accepts either images or "xylists" (xyls),
which are just FITS BINTABLE files which contain two columns (float or
double (E or D) format) which list the pixel coordinates of sources
(stars, etc) in the image.

To specify the column names (eg, "XIMAGE" and "YIMAGE"):
  $ solve-field --x-column XIMAGE --y-column YIMAGE ...

Our solver assumes that the sources are listed in order of brightness,
with the brightest sources first.  If your files aren't sorted, you
can specify a column by which the file should be sorted.
  $ solve-field --sort-column FLUX ...

By default it sorts with the largest value first (so it works
correctly if the column contains FLUX values), but you can reverse
that by:
  $ solve-field --sort-ascending --sort-column MAG ...

When using xylists, you should also specify the original width and
height of the image, in pixels:
  $ solve-field --width 2000 --height 1500 ...

Alternatively, if the FITS header contains "IMAGEW" and "IMAGEH" keys,
these will be used.

The solver can deal with multi-extension xylists; indeed, this is a
convenient way to solve a large number of fields at once.  You can
tell it which extensions it should solve by:
  $ solve-field --fields 1-100,120,130-200

(Ranges of fields are inclusive, and the first FITS extension is 1, as
per the FITS standard.)

Unfortunately, the plotting code isn't smart about handling multiple
fields, so if you're using multi-extension xylists you probably want
to turn off plotting:
  $ solve-field --no-plots ...


BACKEND CONFIG
--------------
Because we also operate a web service using most of the same software,
the local version of the solver is a bit more complicated than it
really needs to be.  The "solve-field" program takes your input files,
does source extraction on them to produce an "xylist" -- a FITS
BINTABLE of source positions -- then takes the information you
supplied about your fields on the command-line and adds FITS headers
encoding this information.  We call this file an "augmented xylist";
we use the filename suffix ".axy".  "solve-field" then calls the
"backend" program, passing it your axy file.  "backend" reads a config
file (by default /usr/local/astrometry/etc/backend.cfg) that describes
things like where to find index files, whether to load all the indexes
at once or run them one at a time, how long to spend on each field,
and so on.  If you want to force only a certain set of indexes to
load, you can copy the backend.cfg file to a local version and change
the list of indexes that are loaded, and then tell solve-field to use
this config file:

   $ solve-field --backend-config mybackend.cfg ...


SOURCE EXTRACTION USING SEXTRACTOR
----------------------------------
http://terapix.iap.fr/rubrique.php?id_rubrique=91/

The "Source Extractor" aka "SExtractor" program by Emmanuel Bertin can
be used to do source extraction if you don't want to use our own
bundled "image2xy" program.

You can use the following configuration entries to produce an "xylist"
which can be fed to solve-field:

   CATALOG_TYPE     FITS_1.0
   # this is the output filename:
   CATALOG_NAME     out.xyls
   PARAMETERS_NAME  xylist.param

In the file "xylist.param", include:

   X_IMAGE
   Y_IMAGE
   MAG_ISO
   # or MAG_ISOCOR or MAG_AUTO

After running SExtractor on your image, you can feed the resulting
file straight to solve-field:

   $ solve-field out.xyls --x-column X_IMAGE --y-column Y_IMAGE \
          --sort-column MAG_ISO --sort-ascending --overwrite \
          --width 496 --height 340


WORKAROUNDS
-----------
*** No python

There are two places we use python: handling images, and filtering FITS files.

You can avoid the image-handling code by doing source extraction
yourself; see the "No netpbm" section below.

You can avoid filtering FITS files by using the "--no-fits2fits"
option to solve-field.

*** No netpbm

We use the netpbm tools (jpegtopnm, pnmtofits, etc) to convert from
all sorts of image formats to PNM and FITS.

If you don't have these programs installed, you must do source
extraction yourself and use "xylists" rather than images as the input
to solve-field.  See SEXTRACTOR and XYLIST sections above.

ERROR MESSAGES during compiling
-------------------------------

1.    /bin/sh: line 1: /dev/null: No such file or directory

We've seen this happen on Macs a couple of times.  Reboot and it goes
away...

2.    makefile.deps:40: deps: No such file or directory

Not a problem.  We use automatic dependency tracking: "make" keeps
track of which source files depend on which other source files.  These
dependencies get stored in a file named "deps"; when it doesn't exist,
"make" tries to rebuild it, but not before printing this message.

3.    gnu-specific-test.c: In function 'main':
      gnu-specific-test.c:23: warning: implicit declaration of function 'canonicalize_file_name'
      gnu-specific-test.c:23: warning: initialization makes pointer from integer without a cast
      /usr/bin/ld: Undefined symbols:
      _canonicalize_file_name
      collect2: ld returned 1 exit status

Not a problem.  We provide replacements for a couple of GNU-specific
functions, but we need to decide whether to use them or not.  We do
that by trying to build a test program and checking whether it works.
This failure tells us your OS doesn't provide the
canonicalize_file_name() function, so we plug in a replacement.

4.    configure: WARNING: cfitsio: == No acceptable f77 found in $PATH
      configure: WARNING: cfitsio: == Cfitsio will be built without Fortran wrapper support
      drvrfile.c: In function 'file_truncate':
      drvrfile.c:360: warning: implicit declaration of function 'ftruncate'
      drvrnet.c: In function 'http_open':
      drvrnet.c:300: warning: implicit declaration of function 'alarm'
      drvrnet.c: In function 'http_open_network':
      drvrnet.c:810: warning: implicit declaration of function 'close'
      drvrsmem.c: In function 'shared_cleanup':
      drvrsmem.c:154: warning: implicit declaration of function 'close'
      group.c: In function 'fits_get_cwd':
      group.c:5439: warning: implicit declaration of function 'getcwd'
      ar: creating archive libcfitsio.a

Not a problem; these errors come from cfitsio and we just haven't
fixed them.


LICENSE
-------

The Astrometry.net code suite is free software licensed under the GNU
GPL, version 2.  See the file LICENSE for the full terms of the GNU
GPL.

The index files come with their own license conditions.  See the file
GETTING-INDEXES for details.

CONTACT
-------
Please send inquires and bug reports to code@astrometry.net .