Recently in Data Integration Category

Mark Willis comes up with another awesome application of photogrammetry to rock art studies.

Follow this link to the entry on his blog.

Go Matsumoto, a graduate student at Southern Illinois University at Carbondale posted his MA thesis on Academia.edu. The work is entitled, "Pachacamac GIS Project: A Practical Application of Geographic Information Systems and Remote Sensing Techniques in Andean Archaeology". This thesis is a wonderful case example that incorporates the use of archival maps and new data to produce an outstanding GIS of the site of Pachacamac, Luirn Valley, Peru. Click on the title above or click here to download or read this thesis.

Recently I have been hunkerd down building 3D models of large archaeological sites using the application Photoscan to processing images captured by kite aerial photography (KAP). Though I need to submit some posts on this work, I can report that the method works very well. While processing KAP images with PhotoScan, I've been wondering if it is possible to return to images in my archive that were taken during excavation and reconstruct 3D models from them. For example, while excavating at the site of Jiskairumoko we took lots of overhead photographs of the units. We did this with the aim of georeferencing the images and mapping small artifacts scatters from the photographs. That effort in planimetric 2D photogrammetry worked well. However, I've been awake at night wondering if it is possible possible to reconstruct 3D models from these images? Yesterday, I took some time to find out; turns out it works pretty well. Some examples are shown below. These images were captured in 2002.



These images below were captured in 2001 by Nico Tripcevich. These images were taken specifically with the aim of generating a 3D model. Nico made a nice panoramic object model, and these images processed very well in Photoscan.


One thing I would do differently to improve model building, is to ensure that all space within the unit is covered by at least two photographs. In the upper two pictures, I had to use a smooth geometry building routine that filled in some holes. I don't think that the model is a great departure from reality. However, had there been paired images covering the entire excavation, I could have used an exact reconstruction method.

For archaeological and anthropological purposes, it is frequently desirable to transform modeled objects (buildings, mounds, plazas, etc.) into real world coordinates. Doing so properly adjusts the scale, orientation, and location of the model. However, displaying models in raw UTM coordinates can cause problems for computer video cards. This post describes two approaches to a coordinate shift that can solve the problem.

This post describes the process of recording surface artifacts by means of geotagged and hierarchically coded photographs.

Equipment:
Digital Camera
Hand held GPS receiver (optional, for geotagging the photos)
Numbered Targets (orange cones, bright plastic plates, cut piece of plastic, or other equivalently noticeable object)
Mapping Equipment (High accuracy GPS or total station)
Laptop Computer

Software:
Structure from motion image processor (Microsoft's Photosynth or Noah Snavely's Bundler)
Geotagging program optional (Friedemann Schmidt's GeoSetter)
Point cloud viewing and measurement software (Menci's ScanView)
Coordinate transformation software (Graticule 3D)
Text editor (Notepad++)
1.1.
Place numbered targets on the surface or object that is to be recorded.
Because the numbered targets have to be located in the point cloud, it
is advisable to use numbered targets of colors that contrast strongly
with the surface of object that is to be recorded. Because of the way
that the points will be measured later, the first numbered point should
be numbered 0. This is because during the measurement process described
under 2.10 ScanView
begins its automatic numbering at 0. If for some reason, it is
absolutely necessary to begin numbering at 1, the measurements made in ScanView
can be renumbered--but do not forget to do this. When placing the
numbered targets, be sure that they will be visible from a number of
different shooting angles. It is generally useful to make a sketch map
that illustrates the location of the numbered targets.

1.2.
Start a small hand held GPS and keep it running throughout the entire
duration of photo shoot. This GPS data will be used to geotag the
photographs. Note that in order for geotagging to solve properly, the
clocks of the GPS and the camera must be perfectly synchronized. This
calibration of the camera's clock should be performed prior to taking
any photographs.

1.3. Following general guidelines for structure
from motion photography, capture images of the surface or object that
is to be recorded. Ensure that between each of the photographs there
are low angles (<15°) and that overlap is high (>70%). Be sure to
take detailed photographs of the numbered targets. Consider doing this
by zooming in on or shooting them with a long lens. These detailed
shots are required to get good representation of the numbered targets
in the point cloud.

1.4. Map the numbered targets with a survey
instrument. This may be done with either a high accuracy GPS or an
accurately coordinated total station.

1.5. Collect the numbered targets. The fieldwork stage is complete.

2.1 Load the photographs onto the computer.

2.2.
If geotagging the images, download the GPS data from the hand held
receiver. Geotag the photographs and apply any additional information
to the EXIF headers (i.e. who took the photographs). GeoSetter is a free program that can be used to geotag the images.

2.3. Downsize the photographs to the appropriate dimensions for the given structure from motion processor that will be used. Photosynth downsamples images to 2 megapixels. Bundler has processed images as large as 3.8 megapixels. Determine the megapixel dimensions by multiplying the height by the width. Megapixel = one million pixels.

2.4. Submit the downsampled image collection to the structure from motion processor. If using Photosynth, simply load the images into the Photosynth interface and begin processing. If using Bundler, there is additional information on installation and image processing here.

2.5.
From the high accuracy GPS or total station, download to the computer
the files that contain the mapped location of the numbered targets.

2.6.
Generate a text file that contains the targets with their corresponding
numbers. Do not include a header row. The format of the text file
should be as follows:
ID X Y Z

2.7. Once the structure from motion processor is complete obtain the point cloud files. If the images were processed with Photosynth, then obtain the point cloud by using the free Photosynth Point Cloud Exporter. When using this tool, export the point cloud as an ASCII ply file. If the images were processed with Bundler,
then look in the bundle directory and copy the ply file with the
highest number. Past this file into the directory where other files
related to the model are stored (i.e. the directory that contains the
text file with the mapped coordinates of the numbered targets).

2.8. From the ply file, delete the header information. All that should remain are six columns of data in the following format:
X Y Z R G B

2.9. After removing the header information, save the file as a text file.

2.10. Using the free program ScanView,
open inspect the text file that contains the point cloud information.
Explore the point cloud until all, or most, of the numbered targets are
located. Look for the groups of points that represent the numbered
targets. Once the numbered targets are located, use the ScanView measure tool to obtain the XYZ values for the appropriate point. Do this for each of the numbered targets.

2.11.
In the Scan View Measure dialog box, select all of the measured points
and click the copy button. Open a text editor, like the free Notepad++,
and select paste. This will produce a tab delimited array according to
the following format:
ID X Y Z

2.12. Save this file that contains the modeled coordinates representing the location of the numbered targets.

2.13. Close the point cloud file from ScanView, and then open this file in Excel. The format of the file should be:
X Y Z R G B

2.14.
Add a new column before the X column and add three new columns before
the R column. If B = a blank column, then the file should have the
following format:
B X Y Z B B B R G B

2.15. Above the very
first line of the file, add a new row for each of the numbered targets.
For example, if there are 10 numbered targets then add 10 new rows. In
these new rows, add the information from the file that was saved in
step 2.6, the file that contains the modeled coordinates of the numbered
targets. Add the value 0 into each of the blank columns that are
between the Z and the R column. For each of the numbered targets, add
the value 255 into each of the R G and B columns respectively.

2.16.
Beginning with the first number higher than the highest numbered
target, use the fill functions in Excel to give numbers to each of the
the renaming records that make up the points forming the point cloud.

2.17.
For each of the three empty columns that are between the Z and the R
column, add the value 0, and copy this value to each of the records
that make up the point cloud. Once these edits are made, save two
versions of the file. Where = the name of the file, save
one version as "_proc" and save the other version as
"_RGB". Close both of the files.

2.18. Now the files are properly formatted the "_proc" file will be passed to Java Graticule 3D where the Helmert coordinate transformation will be performed. The "_RGB" will be used later.

2.19.
Open the free program Java Graticule 3D. Click on the menu
"Transformation" and select 3D. A new window will open. This is the
CoordTrans program. From the CoordTrans window, under
"Transformationsarten:" be sure that 3D is selected. Click on the drop
down menu, and select 9-Parametertransformation-3D (Mx, My, Mz, Rx, Ry,
Rz, Tx, Ty, Tz). Click on the "Dateil" menu, select "Import
Startsystem", and browse to the edited point cloud file
"_proc". If the file is large, it may take a few minutes
for the file to load. Be patient. Once the file has loaded, look at the
first few records and ensure that they represent the numbered targets
that were entered earlier.

2.20. Once the "Startsystem" file
"_proc" loads, click the "Datel" menu, select "Import
Zielsystem" and brows to the surveyed control point file created in
step 2.17. The file should load quickly. Ensure that the coordinates for
the mapped numbered targets are present and correct.

2.21. Click
the "Berechnung starten" button and wait for CoordTrans to perform the
transformation. Once the transformation is complete, click on the
Transformation tab at the bottom of the window. Compare the coordinates
of the modeled numbered targets against the coordinates of the mapped
numbered targets. The values should be very close to one another. If
they are, then click the "Datei" menu, select "Export Transformation",
browse to where the file should be saved, name the file properly, and
save the file. The suggested name of the file is "_trans"

2.22.
From Excel, open the exported file "_trans" and open the
file "_RGB". From the file, "_RGB" copy the
colums that contain the RGB information and paste those values into the
file "_trans". The file "_trans" should now
have the form of:
ID X Y Z OX OY OZ R G B

2.23. Add a header
row with the appropriate column names and save the file. If importing
into ArcGIS, save an additional copy of the file as an Excel file.
Close both of the these files. The file "_trans" contains
the properly transformed point cloud. This file can be imported into a
GIS by loading the table and displaying XY data. Remember that northing
is Y and easting is X.

The image below shows a a blue structure
from motion point cloud and a green total station point cloud. Using
the procedures outlined above, the blue structure from motion point
cloud was properly oriented in real world space.

Blender is free and open source 3D editing software. Like MeshLab, Blender provides some convenient tools for editing point clouds that are generated by structure from motion programs such as Noah Snavely's Bundler or Microsoft's Photosynth. While MeshLab does some nice things, it does not presently permit the selection and editing of individual vertecies. Because structure from motion algorithms can generate a considerable amount of noise, being able to edit the files is very important. Enter Blender...

In Blender, navigation and editing can be tricky. To help ease the learning curve for others who might wish to eit point clouds generated through structure from motion algorithms, I've assembled the following information that I've found useful in my own work. This post describes how to: adjust aspects of the 3D View; make selections; and how to transform point clouds (move, rotate, and scale).

Version 9.3 of ArcMap permits you to bring in kml files created in Google Earth. However, if you do this you will find that the features are not accompanied by descriptions or folders. If you have locations organized within Google Earth into various folders, and if they have any descriptions or names that you would like to preserve when bringing into ArcMap, then use the following kml_to_shp script instead, available at: http://arcscripts.esri.com/details.asp?dbid=15603.

Here is the description from the ESRI Support Center website: "This script converts features in a KML file to an ESRI shapefile. KML files can be created by digitizing features in Google Earth. These data must be converted to a shapefile format before they can be used in a GIS. The script provides an easy method to convert these kml features to shapefiles. The shapefile retains the names, descriptions, and the Google Earth folder of the kml features. The script is run out of ArcToolbox and will work with any license level of ArcGIS 9.2 and 9.3. The tool's documentation describes its use. The accompanying word document describes how to load the toolbox into ArcToolbox."

I highly recommend this if you are using Google Earth to digitize information that you intend to use in ArcMap. Many thanks to Carrie Hritz for bringing the script to my attention.

In this post, I'm going to discuss: PhotoSynth, the underlying structure-from-motion technologies (i.e. Bundler), and then describe some of the research potentials that I see arising from the two. I am starting to apply Photosynth and other structure from motion technologies to topographic and archaeological mapping. Related posts include:

• practices for taking the photographs
• strategies for data processing (in preparation).

For now, I'm going to keep this introduction general but take a look at this video that was posted to YouTube. It is a nice proof of concept demonstration of the technologies I discuss.

This post provides updated documentation on communication between a Leica TCR 1105 total station and a laptop computer. Previously, I suggested that formatting the PCMCIA card on a laptop computer would provide a solution for transferring files from the total station to a laptop. This solution worked for a while, but yesterday I experienced a problem in the field that indicates formatting the PCMCIA card with windows is a very bad solution. While shooting points with the total station, I got an error message saying "Memory Failure" and I was asked if I wanted to format the card. I replied no, and attempted to proceed with the job. While the error occurred, I was attempting to position the gun using the orientation and height tranfer function. The error on the memory card would not allow the gun to orient. I eventually had to reformat the card on the total station. This means that all data on the card is lost. Fortunately, I had backed up all data on the card so reformatting was not an issue. However, it easily could have been a major loss, right?

After reformatting the card on the gun, there was no problem with gun orientation and no error warnings arose for the rest of the day. At the end of the workday, I attempted to transfer the files from the PCMCIA card to the laptop computer. While attempting to do this, I got the "Incorrect Function" error and the card would not read. Yes, back to square one and I did not want to reformat the card and lose all of the days work.

Today, I returned to Huacho (Peru) where I had a serial to USB cable stored for possible situations just like this. Previously, I had not been successful in getting this conversion cable to work properly. Today, it thankfully worked. Here is the make of the cable and what I did.

Cable make: Gigaware 6-tf USB-A to Serial Cable. It came with Driver CD 26-949. The cable has a box attached to it, and it is not a "simple" USB to Serial. It is a "fancy" one that has drivers and accepts some of the communication issues that Serial cables require. I strongly suggest having one of these or a similar cable up one's sleeve. Here is the URL:
http://www.radioshack.com/product/index.jsp?productId=3120513


Procedures for Windows XP:
1) I installed the drivers for the cable using the provided CD.
2) Then I attached the cable (w/o the gun or anything attached to the cable).
3) I waited for Windows to recognize the cable and to tell me that my new hardware was ready to use. It seems that Windows did an internet search for the driver. Strange considering that I believe the driver was just installed from the CD, but I fear that one may need an internet connection to do this. The lack of an internet connection may have been the reason this cable did not work successfully in Pukara (Peru).
4) I attached the Lemo cable (the one with the five pins, two notches, and the red line) on the Leica to its communication port (the one with the five holes and the two notches).
5) I attached the Serial to USB cable (still plugged in) to the Serial end of the Lemo cable that was just plugged into the Leica.
6) On windows, I opened up the System control panel and went to the Device Manager. There I looked for the RadioShack USB to Serial Cable and made note of which COM port it was using. In this case, the cable used COM5.
7) Then I put the total station into Geocom mode and started up Leica Geo Office.
8) I made sure that the settings on the COM5 port were correctly set in LGO data exchange manager.
9) Browse to the card and retrieve the necessary file.

In the end, I think this is a much better solution than formatting the PCMCIA in Windows. Note: from reading the reviews of the cable, I fear that this solution may not work with either Vista or Win 7--just be aware...

In archaeology it is often useful to represent a surface as a digital elevation model (DEM). With just about any data collection process, there is a tradeoff between the extent of the coverage and the amount of detail that is captured within this coverage. Greater detail takes more time to collect, and smaller spaces can be characterized.

Fieldworkers create DEMs by means of a total station, and this is frequently an efficient method of generating data on surface undulations. However, even with a fast total station it can be extremely time consuming to generate high resolution surface models. Laser scanners may be a useful solution, but these instruments are expensive. Stereo pair photographs can provide a lower cost solution to generating high resolution surface models.

In this post, I describe a workflow that involves processing stereo photographs of a surface to generate a high resolution (50mm) digital elevation model of a space that is roughly 2x4m. The procedure entails generating a dense surface model, or point cloud, in PhotoModeler Scanner. This point cloud is saved as a text file, converted to an Excel spreadsheet, and imported into ArcMap where a DEM is generated.

A new title from the National Academy Press

"As digital technologies are expanding the power and reach of research, they are also raising complex issues. These include complications in ensuring the validity of research data; standards that do not keep pace with the high rate of innovation; restrictions on data sharing that reduce the ability of researchers to verify results and build on previous research; and huge increases in the amount of data being generated, creating severe challenges in preserving that data for long-term use.

Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age examines the consequences of the changes affecting research data with respect to three issues - integrity, accessibility, and stewardship-and finds a need for a new approach to the design and the management of research projects. The report recommends that all researchers receive appropriate training in the management of research data, and calls on researchers to make all research data, methods, and other information underlying results publicly accessible in a timely manner. The book also sees the stewardship of research data as a critical long-term task for the research enterprise and its stakeholders. Individual researchers, research institutions, research sponsors, professional societies, and journals involved in scientific, engineering, and medical research will find this book an essential guide to the principles affecting research data in the digital age."

Notes on integrating data from a Leica Total Station into ArcGIS.

Within ArcToolbox there are two tools for projecting shapefiles.

If the projection is not defined use Define Projection (Management)
When transforming from one projection to another use Project (Management)

Problem: The inability to georeference a raster in ArcMap is likely caused by the fact that the target raster is in a different projection than the active map.

Solution: Use ArcToolbox to reproject the raster into the current projection. Either select the projection from the list or import it from one of the other properly projected layers.

This post contains information on how to download and integrate National Elevation Database digital elevation models (DEMs) into ArcMap.

This post describes how to create new points based on the input of a coordinate pair. The process might be useful for plotting individual sites or other observations. Note, rather than reading in a text file of X,Y coordinates the procedures described in this post involve manual coordinate input of data via a digitizing interface.