root/trunk/src/astrometry/README

Revision 9987, 22.5 kB (checked in by dstn, 2 weeks ago)

add redhat notes

Line 
1Astrometry.net code release 0.2
2===============================
3
4Copyright 2006-2008 Michael Blanton, David W. Hogg, Dustin Lang, Keir
5Mierle and Sam Roweis.
6
7This code is meant as an accompanyment for an upcoming paper and its
8main purpose is to allow reproduction of the results in that paper.
9
10This is research code only and is not intended for scientific use in
11reducing data or other analyses.
12
13This release includes a snapshot of all of the components of our
14current research code, including routines to:
15  * Convert raw USNO-B and Tycho2 into FITS format for easier use
16  * Uniformize, deduplicate, and cut the FITSified catalogs
17  * Build indexes from these cuts
18  * Solve the astrometry of images using these indexes
19
20The code includes:
21  * A simple but powerful HEALPIX implementation
22  * The QFITS library with several modifications
23  * libkd, a compact and high-performance kdtree library
24
25In principle, the code in the tarball is sufficient to reproduce our
26entire astrometry system and replicate our blind astrometry results
27(e.g. on the SDSS fields).  However, this requires the entire USNO-B
281.0 catalogue which is about 80GB in size.  For ease of use, we are
29distributing our pre-cooked indexes (about 25 GB total), which are the
30same files we're using for our web service.
31
32INSTALLING
33----------
34
35To compile everything and run with full functionality, you will need:
36  * gcc
37  * cairo
38  * netpbm
39  * libpng
40  * libjpeg
41  * python (probably >= 2.4)
42  * numpy
43
44-> On Ubuntu or Debian-like systems, the following should suffice:
45
46    $ sudo apt-get install libcairo2-dev libnetpbm10-dev netpbm \
47                           libpng12-dev libjpeg-dev python-numpy
48
49-> On RHEL, you'll need something like:
50    $ sudo yum install cairo.x86_64 cairo-devel.x86_64 netpbm.x86_64 \
51                       netpbm-devel.x86_64 fontconfig-devel.x86_64 \
52                       libXrender-devel.x86_64 xorg-x11-proto-devel.x86_64
53
54If you don't have and can't get these libraries, you should still be
55able to compile and use the core parts of the solver, but you will
56miss out on some eye-candy.
57
58Grab the code:
59  $ wget http://astrometry.net/downloads/astrometry.net-0.2.tar.bz2
60  $ tar xjf astrometry.net-0.2.tar.bz2
61  $ cd astrometry.net-0.2
62
63Build the solving system:
64  $ make
65
66If you installed the libraries listed above, build the plotting code:
67  $ make extra
68
69Install it:
70  $ make install
71
72You might see some error message during compilation; see the section
73ERROR MESSAGES below for fixes to common problems.
74
75By default it will be installed in  /usr/local/astrometry .
76You can override this by either:
77     * editing the top-level Makefile (look for INSTALL_DIR)
78  or * defining INSTALL_DIR on the command-line:
79        For bash shell:
80          $ export INSTALL_DIR=/path/to/astrometry
81          $ make install
82        or
83          $ INSTALL_DIR=/path/to/astrometry make install
84        For tcsh shell:
85          $ setenv INSTALL_DIR /path/to/astrometry
86          $ make install
87
88The astrometry solver is composed of several executables.  You may
89want to add the INSTALL_DIR/bin directory to your path:
90   For bash shell:
91     $ export PATH="$PATH:/usr/local/astrometry/bin"
92   For tcsh shell:
93     $ setenv PATH "$PATH:/usr/local/astrometry/bin"
94
95
96GETTING INDEXES
97---------------
98Next, grab some pre-cooked indexes.  See the file GETTING-INDEXES for
99details about how to do this.
100
101(If you have indexes from a previous version of the Astrometry.net
102software, see the "Upgrading" section below.)
103
104Each index is designed to solve images within a narrow range of
105scales.  The indexes designed to solve small (angular size) images are
106rather large files, so you probably only want to grab the indexes
107required for the images you wish to solve.  If you grab extra indexes,
108the solver will run more slowly, but the results should be the same.
109
110Each index contains a large number of "skymarks" (landmarks for the
111sky) that allow our solver to identify your images.  The skymarks
112contained in each index have sizes (diameters) within a narrow range.
113You probably want to download indexes whose quads are, say, 10% to
114100% of the sizes of the images you want to solve.
115
116For example, let's say you have some 1-degree square images.  You
117should grab indexes that contain skymarks of size 0.1 to 1 degree, or
1186 to 60 arcminutes.  Referring to the table below, you should grab
119indexes 203 through 209.  You might find that the same number of
120fields solve, and faster, using just one or two of the indexes in the
121middle of that range - in our example you might try 205, 206 and 207.
122
123For reference, we used index 202 alone for our SDSS tests (13x9 arcmin
124fields).
125
126---------------------------------------------------------------
127Index Filename          Range of skymark diameters    File Size
128                               (arcminutes)              (MB)
129---------------------------------------------------------------
130index-210+.tar.bz2           60  - 2000                    29
131index-209.tar.bz2            42  -   60                    30
132index-208.tar.bz2            30  -   42                    61
133index-207.tar.bz2            22  -   30                   125
134index-206.tar.bz2            16  -   22                   255
135index-205.tar.bz2            11  -   16                   526
136index-204.tar.bz2             8  -   11                 1,023
137index-203.tar.bz2             5.6  -  8.0               2,089
138index-202.tar.bz2             4.0  -  5.6               4,260
139index-201.tar.bz2             2.8  -  4.0               5,878
140index-200.tar.bz2             2.0  -  2.8              10,058
141--------------------------------------------------------------
142
143Download the indexes you need and put them in the top-level
144(astrometry-0.2) source directory.
145
146Install them:
147   $ make install-indexes
148
149(Or you can extract them into the INSTALL_DIR/data directory by hand.)
150
151
152Next, you can (optionally) configure the solver by editing the file
153   INSTALL_DIR/etc/backend.cfg
154
155
156UPGRADING INDEX FILES
157---------------------
158
159The format of the index files has changed from the 0.1 and 0.11
160releases.  There is a program for updating them, and there is also a
161makefile target.
162
163  $ make upgrade-indexes
164
165If you want to do it by hand, use the program "fix-bb".  Run it on
166each of the ckdt.fits and skdt.fits files in your INSTALL_DIR/data
167directory.
168
169
170SOLVING
171-------
172
173Finally, solve some fields.
174
175(If you didn't build the plotting commands, add "--no-plots" to the
176command lines below.)
177
178If you have any of indexes 213 to 218:
179   $ solve-field --scale-low 10 demo/apod4.jpg
180
181If you have index 219:
182   $ solve-field --scale-low 30 demo/apod5.jpg
183
184If you have any of indexes 210 to 214:
185   $ solve-field --scale-low 1 demo/apod3.jpg
186
187If you have any of indexes 206 to 211:
188   $ solve-field --scale-low 1 demo/apod2.jpg
189
190If you have any of indexes 203 to 205:
191   $ solve-field apod1.jpg
192
193If you have any of indexes 200 to 203:
194   $ solve-field demo/sdss.jpg
195
196
197Copyrights and credits for the demo images are listed in the file demo/CREDITS .
198
199Note that you can also give solve-field a URL rather than a file as input:
200   $ solve-field --out apod1b http://antwrp.gsfc.nasa.gov/apod/image/0302/ngc2264_croman_c3.jpg
201
202
203If you don't have the netpbm tools (eg jpegtopnm), do this instead:
204
205If you have any of indexes 213 to 218:
206   $ solve-field --scale-low 10 demo/apod4.xyls
207
208If you have index 219:
209   $ solve-field --scale-low 30 demo/apod5.xyls
210
211If you have any of indexes 210 to 214:
212   $ solve-field --scale-low 1 demo/apod3.xyls
213
214If you have any of indexes 206 to 211:
215   $ solve-field --scale-low 1 demo/apod2.xyls
216
217If you have any of indexes 203 to 205:
218   $ solve-field demo/apod1.xyls
219
220If you have any of indexes 200 to 203:
221   $ solve-field demo/sdss.xyls
222
223
224OUTPUT FILES
225------------
226
227   <base>-ngc.png   : an annotation of the image.
228   <base>.wcs       : a FITS WCS header for the solution.
229   <base>.new       : a new FITS file containing the WCS header.
230   <base>-objs.png  : a plot of the sources (stars) we extracted from
231                      the image.
232   <base>-indx.png  : sources (red), plus stars from the index (green),
233                      plus the skymark ("quad") used to solve the
234                      image.
235   <base>-indx.xyls : a FITS BINTABLE with the pixel locations of
236                      stars from the index.
237   <base>.rdls      : a FITS BINTABLE with the RA,Dec of sources we
238                      extracted from the image.
239   <base>.axy       : a FITS BINTABLE of the sources we extracted, plus
240                      headers that describe the job (how the image is
241                      going to be solved).
242   <base>.solved    : exists and contains (binary) 1 if the field solved.
243   <base>.match     : a FITS BINTABLE describing the quad match that
244                      solved the image.
245   <base>.kmz       : (optional) KMZ file for Google Sky-in-Earth.  You need
246                      to have "wcs2kml" in your PATH.  See
247                       http://code.google.com/p/wcs2kml/downloads/list
248                       http://code.google.com/p/google-gflags/downloads/list
249
250
251TRICKS AND TIPS
252---------------
253
254*** To lower the CPU time limit before giving up:
255
256    $  solve-field --cpu-limit 30 ...
257
258will make it give up after 30 seconds.
259
260(Note, however, that the "backend" configuration file (backend.cfg)
261puts a limit on the CPU time that is spent on an image; solve-field
262can reduce this but not increase it.)
263
264
265*** Scale of the image: if you provide bounds (lower and upper limits)
266on the size of the image you are trying to solve, solving can be much
267faster.  In the last examples above, for example, we specified that
268the field is at least 30 degrees wide: this means that we don't need
269to search for matches in the indexes that contain only tiny skymarks.
270
271Eg, to specify that the image is between 1 and 2 degrees wide:
272
273   $ solve-field --scale-units degwidth --scale-low 1 --scale-high 2 ...
274
275If you know the pixel scale instead:
276
277   $ solve-field --scale-units arcsecperpix \
278       --scale-low 0.386 --scale-high 0.406 ...
279
280When you tell solve-field the scale of your image, it uses this to
281decide which indexes to try to use to solve your image; each index
282contains quads whose scale is within a certain range, so if these
283quads are too big or too small to be in your image, there is no need
284to look in that index.  It is also used while matching quads: a small
285quad in your image is not allowed to match a large quad in the index
286if such a match would cause the image scale to be outside the bounds
287you specified.  However, all these checks are done before computing a
288best-fit WCS solution and polynomial distortion terms, so it is
289possible (though rare) for the final solution to fall outside the
290limits you specified.  This should only happen when the solution is
291correct, but you gave incorrect inputs, so you shouldn't be
292complaining! :)
293
294
295*** Guess the scale: solve-field can try to guess your image's scale
296from a number of different FITS header values.  When it's right, this
297often speeds up solving a lot, and when it's wrong it doesn't cost
298much.  Enable this with:
299
300   $ solve-field --guess-scale ...
301
302*** Depth.  The solver works by looking at sources in your image,
303starting with the brightest.  It searches for all "skymarks" that can
304be built from the N brightest stars before considering star N+1.  When
305using several indexes, it can be much faster to search for many
306skymarks in one index before switching to the next one.  This flag
307lets you control when the solver switches between indexes.  It also
308lets you control how much effort the solver puts in before giving up -
309by default it looks at all the sources in your image, and usually
310times out before this finishes.
311
312Eg, to first look at sources 1-20 in all indexes, then sources 21-30
313in all indexes, then 31-40:
314
315    $ solve-field --depth 20,30,40 ...
316 or $ solve-field --depth 1-20 --depth 21-30 --depth 31-40 ...
317
318Sources are numbered starting at one, and ranges are inclusive.  If
319you don't give a lower limit, it will take 1 + the previous upper
320limit.  To look at a single source, do
321
322    $ solve-field --depth 42-42 ...
323
324
325*** Our source extractor sometimes estimates the background badly, so
326by default we sort the stars by brightness using a compromise between
327the raw and background-subtracted flux estimates.  For images without
328much nebulosity, you might find that using the background-subtracted
329fluxes yields faster results.  Enable this by:
330
331    $ solve-field --resort ...
332
333
334*** If you've got big images: you might want to downsample them before
335doing source extraction:
336
337   $ solve-field --downsample 2 ...
338or $ solve-field --downsample 4 ...
339or etc
340
341
342*** When solve-field processes FITS files, it runs them through a
343"sanitizer" which tries to clean up non-standards-compliant images.
344If your FITS files are compliant, this is a waste of time, and you can
345avoid doing it.
346
347   $ solve-field --no-fits2fits ...
348
349
350*** When solve-field processes FITS images, it looks for an existing
351WCS header.  If one is found, it tries to verify that header before
352trying to solve the image blindly.  You can prevent this with:
353
354   $ solve-field --no-verify ...
355
356Note that currently solve-field only understands a small subset of
357valid WCS headers: essentially just the TAN projection with a CD
358matrix (not CROT).
359
360
361*** If you don't want the plots to be produced:
362
363   $ solve-field --no-plots ...
364
365
366*** "I know where my image is to within 1 arcminute, how can I tell
367solve-field to only look there?"
368
369In this release of the software, you can't.  We've got plans for v0.3,
370but don't hold your breath!  There is one thing you can do: if you
371know that your image is within one HEALPix
372(http://healpix.jpl.nasa.gov/), and if you're using the
373small-angular-scale indexes with filenames like index-###-##.*.fits,
374then each of these indexes covers one HEALPix (the second number is
375the HEALPix number); you can edit the "backend.cfg" config file to
376tell the "backend" program to only load the index that contains your
377image.
378
379
380*** To convert a list of pixel coordinates to RA,Dec coordinates:
381
382   $ wcs-xy2rd -w wcs-file -i xy-list -o radec-list
383
384Where xy-list is a FITS BINTABLE of the pixel locations of sources;
385recall that FITS specifies that the center of the first pixel is pixel
386coordinate (1,1).
387
388
389*** To convert from RA,Dec to pixels:
390
391   $ wcs-rd2xy -w wcs-file -i radec-list -o xy-list
392
393
394*** To make cool overlay plots: see plotxy, plot-constellations.
395
396
397*** To change the output filenames when processing multiple input
398files: each of the output filename options listed below can include
399"%s", which will be replaced by the base output filename.  (Eg, the
400default for --wcs is "%s.wcs").  If you really want a "%" character in
401your output filename, you have to put "%%".
402
403Outputs include: --new-fits, --kmz, --solved, --cancel, --match,
404--rdls, --corr, --wcs --keep-xylist --pnm
405Also included: --solved-in, --verify
406
407
408*** Reusing files between runs:
409
410The first time you run solve-field, save the source extraction
411results:
412   $ solve-field --keep-xylist %s.xy input.fits ...
413
414On subsequent runs, instead of using the original input file, use the
415saved xylist instead.  Also add --continue to overwrite any output
416file that already exists.
417   $ solve-field input.xy --no-fits2fits --continue ...
418
419To skip previously solved inputs (note that this assumes single-HDU
420inputs):
421   $ solve-field --skip-solved ...
422
423
424"I wanna go fast!"
425------------------
426
427Sadly, gcc doesn't have a flag --super-fast to generate the fastest
428possible code for the machine on which it is running.  Also, we are
429shipping the code with assertion and debugging turned on -- this helps
430ensure that bugs are caught and reported, but it does slow down the
431code.  Here are some things you can do to make the code run faster:
432
433  * in util/makefile.common, set RICKY_BOBBY to yes.
434
435  * we try to guess "-mtune" settings that will work for you; if we're
436    wrong, you can set the environment variable ARCH_FLAGS before
437    compiling:
438
439      $ ARCH_FLAGS="-mtune=nocona" make
440
441    You can find details in the gcc manual:
442      http://gcc.gnu.org/onlinedocs/
443
444    You probably want to look in the section:
445      "GCC Command Options"
446         -> "Hardware Models and Configurations"
447             -> "Intel 386 and AMD x86-64 Options"
448
449    http://gcc.gnu.org/onlinedocs/gcc-4.3.0/gcc/i386-and-x86_002d64-Options.html#i386-and-x86_002d64-Options
450
451XYLISTS
452-------
453
454The solve-field program accepts either images or "xylists" (xyls),
455which are just FITS BINTABLE files which contain two columns (float or
456double (E or D) format) which list the pixel coordinates of sources
457(stars, etc) in the image.
458
459To specify the column names (eg, "XIMAGE" and "YIMAGE"):
460  $ solve-field --x-column XIMAGE --y-column YIMAGE ...
461
462Our solver assumes that the sources are listed in order of brightness,
463with the brightest sources first.  If your files aren't sorted, you
464can specify a column by which the file should be sorted.
465  $ solve-field --sort-column FLUX ...
466
467By default it sorts with the largest value first (so it works
468correctly if the column contains FLUX values), but you can reverse
469that by:
470  $ solve-field --sort-ascending --sort-column MAG ...
471
472When using xylists, you should also specify the original width and
473height of the image, in pixels:
474  $ solve-field --width 2000 --height 1500 ...
475
476Alternatively, if the FITS header contains "IMAGEW" and "IMAGEH" keys,
477these will be used.
478
479The solver can deal with multi-extension xylists; indeed, this is a
480convenient way to solve a large number of fields at once.  You can
481tell it which extensions it should solve by:
482  $ solve-field --fields 1-100,120,130-200
483
484(Ranges of fields are inclusive, and the first FITS extension is 1, as
485per the FITS standard.)
486
487Unfortunately, the plotting code isn't smart about handling multiple
488fields, so if you're using multi-extension xylists you probably want
489to turn off plotting:
490  $ solve-field --no-plots ...
491
492
493BACKEND CONFIG
494--------------
495Because we also operate a web service using most of the same software,
496the local version of the solver is a bit more complicated than it
497really needs to be.  The "solve-field" program takes your input files,
498does source extraction on them to produce an "xylist" -- a FITS
499BINTABLE of source positions -- then takes the information you
500supplied about your fields on the command-line and adds FITS headers
501encoding this information.  We call this file an "augmented xylist";
502we use the filename suffix ".axy".  "solve-field" then calls the
503"backend" program, passing it your axy file.  "backend" reads a config
504file (by default /usr/local/astrometry/etc/backend.cfg) that describes
505things like where to find index files, whether to load all the indexes
506at once or run them one at a time, how long to spend on each field,
507and so on.  If you want to force only a certain set of indexes to
508load, you can copy the backend.cfg file to a local version and change
509the list of indexes that are loaded, and then tell solve-field to use
510this config file:
511
512   $ solve-field --backend-config mybackend.cfg ...
513
514
515SOURCE EXTRACTION USING SEXTRACTOR
516----------------------------------
517http://terapix.iap.fr/rubrique.php?id_rubrique=91/
518
519The "Source Extractor" aka "SExtractor" program by Emmanuel Bertin can
520be used to do source extraction if you don't want to use our own
521bundled "image2xy" program.
522
523You can use the following configuration entries to produce an "xylist"
524which can be fed to solve-field:
525
526   CATALOG_TYPE     FITS_1.0
527   # this is the output filename:
528   CATALOG_NAME     out.xyls
529   PARAMETERS_NAME  xylist.param
530
531In the file "xylist.param", include:
532
533   X_IMAGE
534   Y_IMAGE
535   MAG_ISO
536   # or MAG_ISOCOR or MAG_AUTO
537
538After running SExtractor on your image, you can feed the resulting
539file straight to solve-field:
540
541   $ solve-field out.xyls --x-column X_IMAGE --y-column Y_IMAGE \
542          --sort-column MAG_ISO --sort-ascending --overwrite \
543          --width 496 --height 340
544
545
546WORKAROUNDS
547-----------
548*** No python
549
550There are two places we use python: handling images, and filtering FITS files.
551
552You can avoid the image-handling code by doing source extraction
553yourself; see the "No netpbm" section below.
554
555You can avoid filtering FITS files by using the "--no-fits2fits"
556option to solve-field.
557
558*** No netpbm
559
560We use the netpbm tools (jpegtopnm, pnmtofits, etc) to convert from
561all sorts of image formats to PNM and FITS.
562
563If you don't have these programs installed, you must do source
564extraction yourself and use "xylists" rather than images as the input
565to solve-field.  See SEXTRACTOR and XYLIST sections above.
566
567ERROR MESSAGES during compiling
568-------------------------------
569
5701.    /bin/sh: line 1: /dev/null: No such file or directory
571
572We've seen this happen on Macs a couple of times.  Reboot and it goes
573away...
574
5752.    makefile.deps:40: deps: No such file or directory
576
577Not a problem.  We use automatic dependency tracking: "make" keeps
578track of which source files depend on which other source files.  These
579dependencies get stored in a file named "deps"; when it doesn't exist,
580"make" tries to rebuild it, but not before printing this message.
581
5823.    gnu-specific-test.c: In function 'main':
583      gnu-specific-test.c:23: warning: implicit declaration of function 'canonicalize_file_name'
584      gnu-specific-test.c:23: warning: initialization makes pointer from integer without a cast
585      /usr/bin/ld: Undefined symbols:
586      _canonicalize_file_name
587      collect2: ld returned 1 exit status
588
589Not a problem.  We provide replacements for a couple of GNU-specific
590functions, but we need to decide whether to use them or not.  We do
591that by trying to build a test program and checking whether it works.
592This failure tells us your OS doesn't provide the
593canonicalize_file_name() function, so we plug in a replacement.
594
5954.    configure: WARNING: cfitsio: == No acceptable f77 found in $PATH
596      configure: WARNING: cfitsio: == Cfitsio will be built without Fortran wrapper support
597      drvrfile.c: In function 'file_truncate':
598      drvrfile.c:360: warning: implicit declaration of function 'ftruncate'
599      drvrnet.c: In function 'http_open':
600      drvrnet.c:300: warning: implicit declaration of function 'alarm'
601      drvrnet.c: In function 'http_open_network':
602      drvrnet.c:810: warning: implicit declaration of function 'close'
603      drvrsmem.c: In function 'shared_cleanup':
604      drvrsmem.c:154: warning: implicit declaration of function 'close'
605      group.c: In function 'fits_get_cwd':
606      group.c:5439: warning: implicit declaration of function 'getcwd'
607      ar: creating archive libcfitsio.a
608
609Not a problem; these errors come from cfitsio and we just haven't
610fixed them.
611
612
613LICENSE
614-------
615
616The Astrometry.net code suite is free software licensed under the GNU
617GPL, version 2.  See the file LICENSE for the full terms of the GNU
618GPL.
619
620The index files come with their own license conditions.  See the file
621GETTING-INDEXES for details.
622
623CONTACT
624-------
625Please send inquires and bug reports to code@astrometry.net .
626
Note: See TracBrowser for help on using the browser.