Astrometry.net code release 0.2 =============================== Copyright 2006-2008 Michael Blanton, David W. Hogg, Dustin Lang, Keir Mierle and Sam Roweis. This code is meant as an accompanyment for an upcoming paper and its main purpose is to allow reproduction of the results in that paper. This is research code only and is not intended for scientific use in reducing data or other analyses. This release includes a snapshot of all of the components of our current research code, including routines to: * Convert raw USNO-B and Tycho2 into FITS format for easier use * Uniformize, deduplicate, and cut the FITSified catalogs * Build indexes from these cuts * Solve the astrometry of images using these indexes The code includes: * A simple but powerful HEALPIX implementation * The QFITS library with several modifications * libkd, a compact and high-performance kdtree library In principle, the code in the tarball is sufficient to reproduce our entire astrometry system and replicate our blind astrometry results (e.g. on the SDSS fields). However, this requires the entire USNO-B 1.0 catalogue which is about 80GB in size. For ease of use, we are distributing our pre-cooked indexes (about 25 GB total), which are the same files we're using for our web service. INSTALLING ---------- To compile everything and run with full functionality, you will need: * gcc * cairo * netpbm * libpng * libjpeg * python (probably >= 2.4) * numpy -> On Ubuntu or Debian-like systems, the following should suffice: $ sudo apt-get install libcairo2-dev libnetpbm10-dev netpbm \ libpng12-dev libjpeg-dev python-numpy -> On RHEL, you'll need something like: $ sudo yum install cairo.x86_64 cairo-devel.x86_64 netpbm.x86_64 \ netpbm-devel.x86_64 fontconfig-devel.x86_64 \ libXrender-devel.x86_64 xorg-x11-proto-devel.x86_64 If you don't have and can't get these libraries, you should still be able to compile and use the core parts of the solver, but you will miss out on some eye-candy. Grab the code: $ wget http://astrometry.net/downloads/astrometry.net-0.2.tar.bz2 $ tar xjf astrometry.net-0.2.tar.bz2 $ cd astrometry.net-0.2 Build the solving system: $ make If you installed the libraries listed above, build the plotting code: $ make extra Install it: $ make install You might see some error message during compilation; see the section ERROR MESSAGES below for fixes to common problems. By default it will be installed in /usr/local/astrometry . You can override this by either: * editing the top-level Makefile (look for INSTALL_DIR) or * defining INSTALL_DIR on the command-line: For bash shell: $ export INSTALL_DIR=/path/to/astrometry $ make install or $ INSTALL_DIR=/path/to/astrometry make install For tcsh shell: $ setenv INSTALL_DIR /path/to/astrometry $ make install The astrometry solver is composed of several executables. You may want to add the INSTALL_DIR/bin directory to your path: For bash shell: $ export PATH="$PATH:/usr/local/astrometry/bin" For tcsh shell: $ setenv PATH "$PATH:/usr/local/astrometry/bin" GETTING INDEXES --------------- Next, grab some pre-cooked indexes. See the file GETTING-INDEXES for details about how to do this. (If you have indexes from a previous version of the Astrometry.net software, see the "Upgrading" section below.) Each index is designed to solve images within a narrow range of scales. The indexes designed to solve small (angular size) images are rather large files, so you probably only want to grab the indexes required for the images you wish to solve. If you grab extra indexes, the solver will run more slowly, but the results should be the same. Each index contains a large number of "skymarks" (landmarks for the sky) that allow our solver to identify your images. The skymarks contained in each index have sizes (diameters) within a narrow range. You probably want to download indexes whose quads are, say, 10% to 100% of the sizes of the images you want to solve. For example, let's say you have some 1-degree square images. You should grab indexes that contain skymarks of size 0.1 to 1 degree, or 6 to 60 arcminutes. Referring to the table below, you should grab indexes 203 through 209. You might find that the same number of fields solve, and faster, using just one or two of the indexes in the middle of that range - in our example you might try 205, 206 and 207. For reference, we used index 202 alone for our SDSS tests (13x9 arcmin fields). --------------------------------------------------------------- Index Filename Range of skymark diameters File Size (arcminutes) (MB) --------------------------------------------------------------- index-210+.tar.bz2 60 - 2000 29 index-209.tar.bz2 42 - 60 30 index-208.tar.bz2 30 - 42 61 index-207.tar.bz2 22 - 30 125 index-206.tar.bz2 16 - 22 255 index-205.tar.bz2 11 - 16 526 index-204.tar.bz2 8 - 11 1,023 index-203.tar.bz2 5.6 - 8.0 2,089 index-202.tar.bz2 4.0 - 5.6 4,260 index-201.tar.bz2 2.8 - 4.0 5,878 index-200.tar.bz2 2.0 - 2.8 10,058 -------------------------------------------------------------- Download the indexes you need and put them in the top-level (astrometry-0.2) source directory. Install them: $ make install-indexes (Or you can extract them into the INSTALL_DIR/data directory by hand.) Next, you can (optionally) configure the solver by editing the file INSTALL_DIR/etc/backend.cfg UPGRADING INDEX FILES --------------------- The format of the index files has changed from the 0.1 and 0.11 releases. There is a program for updating them, and there is also a makefile target. $ make upgrade-indexes If you want to do it by hand, use the program "fix-bb". Run it on each of the ckdt.fits and skdt.fits files in your INSTALL_DIR/data directory. SOLVING ------- Finally, solve some fields. (If you didn't build the plotting commands, add "--no-plots" to the command lines below.) If you have any of indexes 213 to 218: $ solve-field --scale-low 10 demo/apod4.jpg If you have index 219: $ solve-field --scale-low 30 demo/apod5.jpg If you have any of indexes 210 to 214: $ solve-field --scale-low 1 demo/apod3.jpg If you have any of indexes 206 to 211: $ solve-field --scale-low 1 demo/apod2.jpg If you have any of indexes 203 to 205: $ solve-field apod1.jpg If you have any of indexes 200 to 203: $ solve-field demo/sdss.jpg Copyrights and credits for the demo images are listed in the file demo/CREDITS . Note that you can also give solve-field a URL rather than a file as input: $ solve-field --out apod1b http://antwrp.gsfc.nasa.gov/apod/image/0302/ngc2264_croman_c3.jpg If you don't have the netpbm tools (eg jpegtopnm), do this instead: If you have any of indexes 213 to 218: $ solve-field --scale-low 10 demo/apod4.xyls If you have index 219: $ solve-field --scale-low 30 demo/apod5.xyls If you have any of indexes 210 to 214: $ solve-field --scale-low 1 demo/apod3.xyls If you have any of indexes 206 to 211: $ solve-field --scale-low 1 demo/apod2.xyls If you have any of indexes 203 to 205: $ solve-field demo/apod1.xyls If you have any of indexes 200 to 203: $ solve-field demo/sdss.xyls OUTPUT FILES ------------ -ngc.png : an annotation of the image. .wcs : a FITS WCS header for the solution. .new : a new FITS file containing the WCS header. -objs.png : a plot of the sources (stars) we extracted from the image. -indx.png : sources (red), plus stars from the index (green), plus the skymark ("quad") used to solve the image. -indx.xyls : a FITS BINTABLE with the pixel locations of stars from the index. .rdls : a FITS BINTABLE with the RA,Dec of sources we extracted from the image. .axy : a FITS BINTABLE of the sources we extracted, plus headers that describe the job (how the image is going to be solved). .solved : exists and contains (binary) 1 if the field solved. .match : a FITS BINTABLE describing the quad match that solved the image. .kmz : (optional) KMZ file for Google Sky-in-Earth. You need to have "wcs2kml" in your PATH. See http://code.google.com/p/wcs2kml/downloads/list http://code.google.com/p/google-gflags/downloads/list TRICKS AND TIPS --------------- *** To lower the CPU time limit before giving up: $ solve-field --cpu-limit 30 ... will make it give up after 30 seconds. (Note, however, that the "backend" configuration file (backend.cfg) puts a limit on the CPU time that is spent on an image; solve-field can reduce this but not increase it.) *** Scale of the image: if you provide bounds (lower and upper limits) on the size of the image you are trying to solve, solving can be much faster. In the last examples above, for example, we specified that the field is at least 30 degrees wide: this means that we don't need to search for matches in the indexes that contain only tiny skymarks. Eg, to specify that the image is between 1 and 2 degrees wide: $ solve-field --scale-units degwidth --scale-low 1 --scale-high 2 ... If you know the pixel scale instead: $ solve-field --scale-units arcsecperpix \ --scale-low 0.386 --scale-high 0.406 ... When you tell solve-field the scale of your image, it uses this to decide which indexes to try to use to solve your image; each index contains quads whose scale is within a certain range, so if these quads are too big or too small to be in your image, there is no need to look in that index. It is also used while matching quads: a small quad in your image is not allowed to match a large quad in the index if such a match would cause the image scale to be outside the bounds you specified. However, all these checks are done before computing a best-fit WCS solution and polynomial distortion terms, so it is possible (though rare) for the final solution to fall outside the limits you specified. This should only happen when the solution is correct, but you gave incorrect inputs, so you shouldn't be complaining! :) *** Guess the scale: solve-field can try to guess your image's scale from a number of different FITS header values. When it's right, this often speeds up solving a lot, and when it's wrong it doesn't cost much. Enable this with: $ solve-field --guess-scale ... *** Depth. The solver works by looking at sources in your image, starting with the brightest. It searches for all "skymarks" that can be built from the N brightest stars before considering star N+1. When using several indexes, it can be much faster to search for many skymarks in one index before switching to the next one. This flag lets you control when the solver switches between indexes. It also lets you control how much effort the solver puts in before giving up - by default it looks at all the sources in your image, and usually times out before this finishes. Eg, to first look at sources 1-20 in all indexes, then sources 21-30 in all indexes, then 31-40: $ solve-field --depth 20,30,40 ... or $ solve-field --depth 1-20 --depth 21-30 --depth 31-40 ... Sources are numbered starting at one, and ranges are inclusive. If you don't give a lower limit, it will take 1 + the previous upper limit. To look at a single source, do $ solve-field --depth 42-42 ... *** Our source extractor sometimes estimates the background badly, so by default we sort the stars by brightness using a compromise between the raw and background-subtracted flux estimates. For images without much nebulosity, you might find that using the background-subtracted fluxes yields faster results. Enable this by: $ solve-field --resort ... *** If you've got big images: you might want to downsample them before doing source extraction: $ solve-field --downsample 2 ... or $ solve-field --downsample 4 ... or etc *** When solve-field processes FITS files, it runs them through a "sanitizer" which tries to clean up non-standards-compliant images. If your FITS files are compliant, this is a waste of time, and you can avoid doing it. $ solve-field --no-fits2fits ... *** When solve-field processes FITS images, it looks for an existing WCS header. If one is found, it tries to verify that header before trying to solve the image blindly. You can prevent this with: $ solve-field --no-verify ... Note that currently solve-field only understands a small subset of valid WCS headers: essentially just the TAN projection with a CD matrix (not CROT). *** If you don't want the plots to be produced: $ solve-field --no-plots ... *** "I know where my image is to within 1 arcminute, how can I tell solve-field to only look there?" In this release of the software, you can't. We've got plans for v0.3, but don't hold your breath! There is one thing you can do: if you know that your image is within one HEALPix (http://healpix.jpl.nasa.gov/), and if you're using the small-angular-scale indexes with filenames like index-###-##.*.fits, then each of these indexes covers one HEALPix (the second number is the HEALPix number); you can edit the "backend.cfg" config file to tell the "backend" program to only load the index that contains your image. *** To convert a list of pixel coordinates to RA,Dec coordinates: $ wcs-xy2rd -w wcs-file -i xy-list -o radec-list Where xy-list is a FITS BINTABLE of the pixel locations of sources; recall that FITS specifies that the center of the first pixel is pixel coordinate (1,1). *** To convert from RA,Dec to pixels: $ wcs-rd2xy -w wcs-file -i radec-list -o xy-list *** To make cool overlay plots: see plotxy, plot-constellations. *** To change the output filenames when processing multiple input files: each of the output filename options listed below can include "%s", which will be replaced by the base output filename. (Eg, the default for --wcs is "%s.wcs"). If you really want a "%" character in your output filename, you have to put "%%". Outputs include: --new-fits, --kmz, --solved, --cancel, --match, --rdls, --corr, --wcs --keep-xylist --pnm Also included: --solved-in, --verify *** Reusing files between runs: The first time you run solve-field, save the source extraction results: $ solve-field --keep-xylist %s.xy input.fits ... On subsequent runs, instead of using the original input file, use the saved xylist instead. Also add --continue to overwrite any output file that already exists. $ solve-field input.xy --no-fits2fits --continue ... To skip previously solved inputs (note that this assumes single-HDU inputs): $ solve-field --skip-solved ... "I wanna go fast!" ------------------ Sadly, gcc doesn't have a flag --super-fast to generate the fastest possible code for the machine on which it is running. Also, we are shipping the code with assertion and debugging turned on -- this helps ensure that bugs are caught and reported, but it does slow down the code. Here are some things you can do to make the code run faster: * in util/makefile.common, set RICKY_BOBBY to yes. * we try to guess "-mtune" settings that will work for you; if we're wrong, you can set the environment variable ARCH_FLAGS before compiling: $ ARCH_FLAGS="-mtune=nocona" make You can find details in the gcc manual: http://gcc.gnu.org/onlinedocs/ You probably want to look in the section: "GCC Command Options" -> "Hardware Models and Configurations" -> "Intel 386 and AMD x86-64 Options" http://gcc.gnu.org/onlinedocs/gcc-4.3.0/gcc/i386-and-x86_002d64-Options.html#i386-and-x86_002d64-Options XYLISTS ------- The solve-field program accepts either images or "xylists" (xyls), which are just FITS BINTABLE files which contain two columns (float or double (E or D) format) which list the pixel coordinates of sources (stars, etc) in the image. To specify the column names (eg, "XIMAGE" and "YIMAGE"): $ solve-field --x-column XIMAGE --y-column YIMAGE ... Our solver assumes that the sources are listed in order of brightness, with the brightest sources first. If your files aren't sorted, you can specify a column by which the file should be sorted. $ solve-field --sort-column FLUX ... By default it sorts with the largest value first (so it works correctly if the column contains FLUX values), but you can reverse that by: $ solve-field --sort-ascending --sort-column MAG ... When using xylists, you should also specify the original width and height of the image, in pixels: $ solve-field --width 2000 --height 1500 ... Alternatively, if the FITS header contains "IMAGEW" and "IMAGEH" keys, these will be used. The solver can deal with multi-extension xylists; indeed, this is a convenient way to solve a large number of fields at once. You can tell it which extensions it should solve by: $ solve-field --fields 1-100,120,130-200 (Ranges of fields are inclusive, and the first FITS extension is 1, as per the FITS standard.) Unfortunately, the plotting code isn't smart about handling multiple fields, so if you're using multi-extension xylists you probably want to turn off plotting: $ solve-field --no-plots ... BACKEND CONFIG -------------- Because we also operate a web service using most of the same software, the local version of the solver is a bit more complicated than it really needs to be. The "solve-field" program takes your input files, does source extraction on them to produce an "xylist" -- a FITS BINTABLE of source positions -- then takes the information you supplied about your fields on the command-line and adds FITS headers encoding this information. We call this file an "augmented xylist"; we use the filename suffix ".axy". "solve-field" then calls the "backend" program, passing it your axy file. "backend" reads a config file (by default /usr/local/astrometry/etc/backend.cfg) that describes things like where to find index files, whether to load all the indexes at once or run them one at a time, how long to spend on each field, and so on. If you want to force only a certain set of indexes to load, you can copy the backend.cfg file to a local version and change the list of indexes that are loaded, and then tell solve-field to use this config file: $ solve-field --backend-config mybackend.cfg ... SOURCE EXTRACTION USING SEXTRACTOR ---------------------------------- http://terapix.iap.fr/rubrique.php?id_rubrique=91/ The "Source Extractor" aka "SExtractor" program by Emmanuel Bertin can be used to do source extraction if you don't want to use our own bundled "image2xy" program. You can use the following configuration entries to produce an "xylist" which can be fed to solve-field: CATALOG_TYPE FITS_1.0 # this is the output filename: CATALOG_NAME out.xyls PARAMETERS_NAME xylist.param In the file "xylist.param", include: X_IMAGE Y_IMAGE MAG_ISO # or MAG_ISOCOR or MAG_AUTO After running SExtractor on your image, you can feed the resulting file straight to solve-field: $ solve-field out.xyls --x-column X_IMAGE --y-column Y_IMAGE \ --sort-column MAG_ISO --sort-ascending --overwrite \ --width 496 --height 340 WORKAROUNDS ----------- *** No python There are two places we use python: handling images, and filtering FITS files. You can avoid the image-handling code by doing source extraction yourself; see the "No netpbm" section below. You can avoid filtering FITS files by using the "--no-fits2fits" option to solve-field. *** No netpbm We use the netpbm tools (jpegtopnm, pnmtofits, etc) to convert from all sorts of image formats to PNM and FITS. If you don't have these programs installed, you must do source extraction yourself and use "xylists" rather than images as the input to solve-field. See SEXTRACTOR and XYLIST sections above. ERROR MESSAGES during compiling ------------------------------- 1. /bin/sh: line 1: /dev/null: No such file or directory We've seen this happen on Macs a couple of times. Reboot and it goes away... 2. makefile.deps:40: deps: No such file or directory Not a problem. We use automatic dependency tracking: "make" keeps track of which source files depend on which other source files. These dependencies get stored in a file named "deps"; when it doesn't exist, "make" tries to rebuild it, but not before printing this message. 3. gnu-specific-test.c: In function 'main': gnu-specific-test.c:23: warning: implicit declaration of function 'canonicalize_file_name' gnu-specific-test.c:23: warning: initialization makes pointer from integer without a cast /usr/bin/ld: Undefined symbols: _canonicalize_file_name collect2: ld returned 1 exit status Not a problem. We provide replacements for a couple of GNU-specific functions, but we need to decide whether to use them or not. We do that by trying to build a test program and checking whether it works. This failure tells us your OS doesn't provide the canonicalize_file_name() function, so we plug in a replacement. 4. configure: WARNING: cfitsio: == No acceptable f77 found in $PATH configure: WARNING: cfitsio: == Cfitsio will be built without Fortran wrapper support drvrfile.c: In function 'file_truncate': drvrfile.c:360: warning: implicit declaration of function 'ftruncate' drvrnet.c: In function 'http_open': drvrnet.c:300: warning: implicit declaration of function 'alarm' drvrnet.c: In function 'http_open_network': drvrnet.c:810: warning: implicit declaration of function 'close' drvrsmem.c: In function 'shared_cleanup': drvrsmem.c:154: warning: implicit declaration of function 'close' group.c: In function 'fits_get_cwd': group.c:5439: warning: implicit declaration of function 'getcwd' ar: creating archive libcfitsio.a Not a problem; these errors come from cfitsio and we just haven't fixed them. LICENSE ------- The Astrometry.net code suite is free software licensed under the GNU GPL, version 2. See the file LICENSE for the full terms of the GNU GPL. The index files come with their own license conditions. See the file GETTING-INDEXES for details. CONTACT ------- Please send inquires and bug reports to code@astrometry.net .