All posts by Ryan Denniston

census, Data Sources, demographics

Welcome to the Current Population Statistics on the Web

Duke University recently acquired access to the online version of its Current Population Statistics (CPS) CD-ROM collection to facilitate easy access to CPS data (Unicon’s CPS Utilities on the Web).  This blog post will walk through the basic data extraction process.  The interface is comparable to that provided by the CD, and users of this collection will find the interface and powerful.  Please note that the instructions provided on the web site are very important to read, particularly for those unfamiliar with the CPS CD version.

Create an Account

When you visit the Unicon site (http://unicon.com/), click the “CPS on Web” link to the left, then click the Register button.  You will have to enter some information to complete the registration process.

Once complete, submit the information.  Once the registration window closes, choose the CPS series (or month) you wish to query, and log in to the system.

 

1Navigation and Data Extraction

Once logged in, you will see a popup window like that shown in the image to the right.  For a typical data extraction, the following steps are advised.

1) First, click the Set Option button and chang4e the timeout to at least 300 seconds.  This will ensure successful data extraction.

2) Next, click the Make an Extraction button, followed by the Request Editor button on the next page.  You should see a page similar to that below (all variables used in your prior extraction will be listed).

23) Remove any variables you do not need.  Next, make certain the variable you wish to include is selected at the top and click “Add Variable(s).”  Alternatively, if you already know the names of the variables, you may type them into the boxes provided on the page.

4) Once all variables are added to the selection, click Continue.  On the following page, specify the output format for the dataset.  Once complete, be certain to select one or more years (at the top).  After you have selected years, click the Extract button.

5) On the following page, you will be presented with a list of variables by year.  As variables change across years in some cases, not all selected variables may be present for each year.  When selecting variables, checking the “View Documentation” checkbox at the top will allow for browsing of available years.

 

Other Useful Tools

– The Make a Table button allows for the construction of crosstabs of observations, means, and other statistics.  This is helpful if the goal is to locate variables for analysis or if there is a choice between two or more variables.

– The Make a Graph button is also useful for data exploration.  The program provides the ability to construct hsitograms, line charts, scatter ploys, pie charts, and bar charts.  Basic summaries of a variable can also be generated from this page.

– If your data need to be weighted to represent the US population, be certain to select the appropriate weight under the Apply Weights button before extraction.

– Subsets of individuals can also be produced under the Specify Universe button.  For example, a specific race or gender can be specified to reduce the sample to what you need.

GIS, tutorial

Online Data Mapping Tools – Google Fusion Tables

Visualizing spatial data can be challenging. Specialized software tools like ArcGIS produce excellent results, but often seem complex for relatively simple tasks. Several online tools have emerged recently that provide relatively easy alternatives for the display of spatial data. In this post, we examine Google Fusion Tables, which combines visualizations, including spatial visualizations, with a database back end. The key advantages to Fusion Tables are easy display of latitude/longitude data or data that is included with address information. In addition, Fusion Tables provides a one-stop location for producing visualizations other than maps, such as line charts or tables.

Uploading Data

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8

Uploaded to Fusion Tables is easy through Google Docs.  Simply log in if you have an account, create a new Table, and on the next screen, point to the file you wish to upload.  Excel and CSV files are the two most commonly used, and KML files allow for upload of maps that contain spatial information, such as locations or polygon definitions.

One thing to note about Google products is that they are often in a state of flux.  Limits and restrictions noted below may change in the future.  For further information regarding Google Fusion Tables, please consult this Libguide authored by Mark Thomas.

Geocoding Addresses

A complete list of geographic data types can be found at the Google support site.  In this post, two of the more common geocoding types will be addressed, address data and data that applies to states, counties, and similar objects.

Address data is pretty easy to work with.  Addresses should contain as much information as possible with items separated by spaces only, no commas.  For example, 134 Chapel Drive Durham NC 27708 should produce a pretty good geocoding result.  In the following example, Durham gun crimes for 2011 were downloaded from the Durham Police Department.  The data only came with address information, so city and state data were subsequently added and combined in Excel (location field).  In Figure 1, highlighted fields indicate spatial information.

To geocode the addresses, select “Map” under the “Visualize” tab.  The program will automatically begin geocoding based on the left-most field containing spatial information, which is city in this case.  Changing the field to “location,” which contains the full address information, will correctly geocode these addresses (Figure 2).

Once complete, the geocoded points are plotted on a map (Figure 3).  As with other mapping applications, you may apply a symbology to the points in order to visualize your data.

This particular dataset contains a numeric field that identifies 5 general types of crime (crime_cat_num).  Under the “Configure styles” link at the top, navigate to “Buckets” and divided the data into five buckets (Figure 4).

Once saved, each color will represent a different type of crime and shown in Figure 5 (red indicates robbery, yellow, assault and so on).

Note that only numeric fields can be used to categorize data, so you may wish to create these fields prior to upload.

Geocoding Polygons

Working with polygon data can be a bit trickier because the polygons must be spatially defined.  Fusion Tables does this by using kml, which is basically a large piece of text containing all of the coordinates, in order, that define a boundary.  For example, in this table, each boundary is defined in the geometry field. Google provides a variety of boundary types, which are available here.  If your data match one of these existing boundary types, you may upload data and merge it with the correct table, which will basically import the boundary definitions into your dataset.  Otherwise, you will have to locate suitable boundaries in a kml file and import those boundaries before merging.

This dataset displays acreage and farms for each county in North Carolina and originally came from the Census of Agriculture.  Note that there must be a field in common between your data and the data containing boundary definitions in order to merge.  In addition, merge fields can only be text fields.  FIPS codes uniquely identify counties and are contained in both tables.  Unfortunately, Google didn’t set up their FIPS fields correctly, so a cleaned up North Carolina county file is located here.

Next, click the merge tab.  Copy and paste the URL for the boundaries table and click the “Get” button.  In Figure 6, I merged  my data to the boundary file using the fips field, which is called “fips” in table 1 and “GEO_ID2” in table 2.  A merge will produce a new table, so be sure to name that new table at the bottom.Once complete, styling the map is comparable to point data. First, select “Map” under the “Visualize” tab, and be sure to point the location field at the top left to “geometry” where the boundary definitions are stored. Next, click the configure styles link. Then, select Fill color under the Polygons section.

In Figure 7, I am showing median farm size (in acres) along a gradient.  It’s important to note the lower and upper limits to your data in advance as the program will not automatically sense this.  In this case, median farm size ranges from 10 to 191.  Figure 8 shows the output.

Sharing Maps

As withsimilar online programs, Fusion Tables allows sharing of data and maps through a variety of avenues, from links to embeddable script to email.  The links below point to the two maps produced in this posting.

Durham gun crimes map

North Carolina median farm acreage, 2007

Other Notes

This overview provides only a brief introduction to the mapping capabilities of Fusion Tables.  A broad gallery of applications is located at this site, and it contains a variety of geography types.  Some of these use the Fusion Tables API, which is a nice feature that allows for application development with some programming experience.  As with the other tools reviewed by this blog, non-standard boundaries are generally absent and can be difficult to locate.  For example, a researcher with country-level data from the 1700s may have difficulty finding a country border map from that time.  However, maps are available on Fusion Tables for counties, states, countries, and congressional districts, and additional maps can be found on the Internet.

GIS

Introducing ArcGIS Online Public Version and Special Access for Duke Users

Overview

ArcGIS Online is a service that allows for storage and sharing of spatial data and maps. In contrast to many other web based GIS services, ArcGIS Online accepts geocoded text-based data and shapefiles, which allows users to share and present work built in ArcGIS Desktop.

Members of the Duke community can register for two different version of ArcOnline. Public access to the service grants access that allows basic file storage and digital mapping. Duke-sponsored access  facilitates sharing files within the Duke community and provides a higher threshold for storing and processing ArcGIS files online. Access to the Duke version is available on request for Duke affiliates with a valid Duke email at askdata@duke.edu.

 

Loading and Processing Data

Duke-sponsored access allows users to import and work with text-based data sets containing more than 250 features or shapefiles containing more than 1,000 features online.  Data sets that exceed these thresholds must be published as Feature Services, which can done at the time of upload or any point thereafter.  A Feature Service is basically an object that can be brought into a map and differs from a file, which can be uploaded for storage, but cannot be imported into a map.

Figure 1: Geolocation in ArcGIS Online

In addition, Duke sponsored access will allow you to use data shared by other members of the Duke community, which expands the data available beyond those data sources shared with the public and by members of groups to which you belong, both available in either version.  Users with modest data needs and users that prefer use ArcGIS Desktop to create maps will find the public version suitable in most cases.  Users with larger datasets and those that will collaborate and present maps online will find Duke-sponsored access much more helpful.ArcGIS Online provide two entry points for the uploading of data and the production of maps.  The first is entered when “My Content” is clicked.  This section lists all of the items that have been uploaded and produced by the user.  There are two key types of items listed, files and data sources.  Files, including text files and shapefiles, are items that can be stored and shared, but are not accessible by the mapping interfaces.  By contrast, data sources, the most common of which are Feature Services and Web Maps, can be seen by the mapping interfaces and incorporated into new maps.

Duke sponsored access provides the ability to convert geocoded text files and shapefiles into data sources at the time of upload or any point thereafter.  By contrast, the public version does not allow users to create Feature Services, but does allow for the creation of Web Maps data sources from text files containing fewer than 250 features or shapefiles containing fewer than 1,000 features.  Data sets exceeding these thresholds will require Duke sponsored access to visualize online.

 

Mapping

The second section of ArcGIS Online is the “Map” section, which opens the map viewer, one of the two mapping tools available in ArcGIS Online.  The map viewer allows for the creation of Web Maps, which can be shared online and saved as data sources for new maps.  Both versions of ArcGIS Online will allow for the upload of data sources directly into the viewer, the inclusion of public and group data to which the user has access, and the inclusion of data sources previously created by the user.

Figure 2: Customized styling in Map Viewer

Once saved, the map can be accessed in the “My Content” section and opened in either map viewer or Explorer, the second mapping tool available.Map viewer allows you to add data from files, web services, and allows for the creation of editable layers.  Many styling modifications like color classification of features and customization of the attribute popup window are possible.  This map displays a customized popup with an added pie chart based on, in this case, a single feature.

Explorer contains the same basic set of features, but it also contains a presentation mode, where slide stills can be taken and arranged for presentation.  Again, this map displays a styled popup as well as customized county-level styling based on an attribute.

 

Sharing Data and Maps

Saved maps can be shared by embeddable script or by link.  The map as a data source can also be shared with the public, with members of any groups to which the user belongs, and with the Duke community as a whole (“Duke University and Medical Center –NSOE”).

Figure 3: Presentation mode in Explorer

 

Conclusion

Online sharing of data and collaboration is a relatively new need that multiple tools are working to fulfill.  ArcGIS Online is an excellent option, particularly when online viewing is an important goal.  If your online visualization needs are modest, and if you generally prefer to produce maps and edit shapefiles on ArcGIS Desktop, the public version may fulfill your needs.  But if the feature restrictions noted above prove prohibitive, Duke sponsored access will provide the flexibility needed for most applications.

Data Visualization, GIS, Uncategorized

Online Mapping Tools – GeoCommons

Visualizing spatial data can be challenging.  Specialized software tools like ArcGIS produce excellent results, but often seem complex for relatively simple tasks. Several online tools have emerged recently that provide relatively easy alternatives for the display of spatial data.  In this post, we examine GeoCommons, a web based tool for presenting spatial data in detail.  (Go to this guide to see a comparison chart of packages and features, and see this Duke University Libguide for a more detailed review of GeoCommons.)

 

GeoCommons (geocommons.com)

GeoCommons is an online mapping application that easily imports a variety of data formats, including geospatial data, and quickly produces sharable maps.  In contrast to other mapping tools, GeoCommons contains several categorization algorithms, such as quantile classification and classification based on the standard deviation of the sample that assist with the construction of informative maps.  CSV files and ArcGIS shapefiles are two of the most widely used file formats compatible with GeoCommons.

GeoCommons is very easy to use and contains some of the display features contained in high-end GIS suites.  Creation of new variables tied to geographies can be tricky, so it’s advised to either upload data  and map in final form or to first identify the layer to which you will upload and join a complete data set.

 

Geocoding

Figure 1

To begin geocoding, upload a file.  Gecommons has the capability to recognize spatially encoded data.  Some formats may require user assistance.

If you’ve uploaded data that contains latitude and longitude coordinates, choose this option.  In my case, I had county FIPS codes that uniquely identified each county.  Selecting US Boundaries to the left, then USA Counties, I was able to successfully preview how well my FIPS codes matched the layer (Figures 1 and 2).  A variety of other boundary types are available.  The key is to have in your data a unique identifier that identifies each record in the same manner as an available geocoding layer.

Figure 2

Review the geocoding results and select Continue to proceed.

 

Mapping

Geocommons offers some nice built in features that assist with categorizing measures.  The application will produce summary statistics for numeric fields (Figure 3), which gives you a quick picture of your sample and can assist with how to categorize the data.  Click the “Make a Map” button to proceed to the interactive interface.

Figure 3

Also note the filter tab, which allows you to screen out groups of cases.  For example, I may request a minimum number of farms to screen out urban counties.

Figure 4 shows a standard choropleth map portraying median number of acres per farm by county for North Carolina in 2007.  In this example, I have classified counties into five groups using standard deviations to group counties.

 

Sharing

Figure 4

GeoCommons contains a wide variety of ways to share data (accessed through the About section).  Posting to Twitter, Facebook, and an array of other social media sites is possible with a few short clicks.  You can directly email a link to the map along with a short personal message right out of the application.For those who wish to post to a web page, GeoCommons provides two ways to insert a map, through a <div> tag and through an iframe.  All code is generated for copy and paste into your page.

To access a version of this map, simply follow this link.

Finally, GeoCommons will produce a PNG image and a KML document for download.  The image export feature appears to be relatively new and does take trial-and-error to align correctly.  In addition, it does not appear to include any base layers or legends in the output, only the data layer.

 

Other Notes

When using standard deviation and maximum breaks methods for grouping observations, double check the category definitions by changing the number of categories and the resulting changes to the definitions for the new groups.  This will help to confirm whether data are grouped appropriately and exactly what the definitions for each category are.

Data Visualization, GIS

Online Mapping Tools – Tableau Public 7

1 Comment

Visualizing spatial data can be challenging to learn. Specialized software tools like ArcGIS produce excellent results, but often seem complex for relatively simple tasks. Several online tools have emerged recently and provide relatively easy alternatives for the display of spatial data. In this ongoing series of alternatives, we review Tableau Public 7 in detail.  Go to this guide to see a comparison chart of packages and features, and see this Duke University Libguide for a more detailed review of Tableau 6.1.

 

Tableau Public (link)

Figure 1

Tableau Public is a free software application that allows you to easily map data and share maps through email or web pages by embeddable script. To use Tableau, you must download and install a free desktop application. Tableau Public also requires a free registration to share visualizations created in the software.

Tableau is designed to look and feel like a standard spreadsheet application. Geographic mapping is accomplished by dragging your coordinate fields and dropping them into the columns and rows fields (see Figure 1). In Tableau 7, you may also select “Filled Map” under the Marks panel, and select a geographic identifier for the “Level of Detail” field (see Figure 2). Once done, add the variable to color by to the color field. In these examples, more intense colors indicate larger median farm size, measured in acres.

Figure 2

 

Geocoding

Tableau generates new fields that hold coordinate data as it imports and geocodes your data. If you wish to create filled maps (states, counties, etc.) in Tableau 7, you must additionally have geographic identifiers that are unique for each case. In Figure 2, the initial map only contained 50 polygons, as 50 North Carolina counties were uniquely named within the United States.

Had I also included a state field, unique identification would have been automatic, but Tableau allowed me to define the state for each case, and lucky for me, I only had North Carolina data.

The geocoding options are extensive. The following list is not exhaustive: area codes, FIPS codes, county/state/country names, ZIP codes, and ISO country codes. Of course, any coordinate data will work for point data.

 

Sharing

Sharing on a web page is accomplished through embeddable Javascript. Sadly, I was unable to get Tableau to work within WordPress, but you may see a live version of this map by following this link.

 

Other Notes

Tableau is very easy to use, provided your data is reasonably clean. With geographic data, be certain to either have something that uniquely identifies each entity or have latitudes and longitudes. It is preferable to err on the side of including more identification fields rather than less (i.e. including state names in addition to counties).

Also be aware that Tableau is not backward-compatible. For example, the workbook used in this example was initially created in Tableau 6.1, modified in Tableau 7, but failed to open once I moved back to Tableau 6.1. However, irrespective of version, you will be able to see any visualizations produced in any version.

 

census, demographics, GIS

ACS Mapping Extension for ArcGIS

The Census Bureau’s American Community Survey provides a continuous measure of the community demographics in the US.   A  new extension provided by the Department of Geography and Geoinformation Science at Geroge Mason University enhances the mapping of ACS by data by allowing researchers to visualize both survey estimates while revealing the level of uncertainty in the estimates.  ACS Mapping Extensions is an ArcGIS addon available for both ArcGIS 9.3 and 10.  This post provides a brief overview of installation, setup, and use.  Detailed technical assistance is provided by the extension.

 

 

Installation
1) Once you download the program, you will want to install and note the installation directory.  In ArcGIS, select Customize from the menu bar, and click Customize Mode….  Then select “Add from file…” and navigate to the installation directory.  Once in this directory, select the “ACSMapping.tlb” file.

 

2) Before you leave the Customize window, be sure to check the “ACS Mapping Tools” toolbar.  You will have a new “ACS Mapping” toolbar added to your window.

 

 

Setup
1) The “Documentation” option in the “ACS Mapping” toolbar provides detailed instructions for downloading ACS data and boundary files.  Follow these instructions to the letter and to their entirety.  With respect to boundary files, the TIGER 2008 county boundaries were used for this example.

 

2) Add the boundary layer to a blank map and select “Join ACS Table(s) with Shapefiles” option in the “ACS Mapping” toolbar.  In this example, I have downloaded county boundaries and county-level median income data from the 2005-09 ACS.  In this figure, the first two fields indicate the items to be joined, one table to one shapefile.   “CNTYIDFP” represents the FIPS code in the boundary file, and “GEO_ID2” is the corresponding code in the ACS table.  Once you’ve set an output location, select “OK.”

 

3) Finally, you will want to apply a symbology to the layer.  In this case, I chose the median income estimate and 5 total categories.  The following figure shows what my map looks like at this point.

 

 

Mapping ACS Estimates with Coefficients of Variation

1) The tools are located under the “Mapping Data Uncertainty” option in the ACS Mapping toolbar.  The first option, “Overlay CVs with Estimates,” will allow you to visualize the uncertainty of estimates at the same time as the estimates themselves.    As noted by the documetation provided by the ACS Mapping Extension web site, ACS provides a margin of error that produces a confidence level of 90%.  This tool will convert these data into coefficients of variation that will allow you to assess the quality of the estimates.

 

2) Select the target layer to whcih you added symbology, select the variable that stores the estimate to be calculated, and finally, select the variable that stores the margin of error (suffix = “_M”).

 

3) After you click the “Select” button, you will be presented with the new Symbology options for the new coefficients of variation layer to be generated.  In this case, I retained the automatic selections and hit “OK.”

 

4) Zooming in to central North Carolina, one can see not only that the Research Triangle Area has relatively high incomes compared with much of North Carolina, but that coefficients of variation are lower than thay are for parts of northern North Carolina and southern Virginia.

 

 

Measuring Singificant Differences in Income
1) The second option, “Identify Areas of Significant Differences,” allows you to assess whether there is a significant difference between one spatial unit and all other spatial units for a given variable.  In order for this option to work, you must select one specific spatial unit.  In this example, I selected Durham County and will assess whether there are significant differences in median household income in the region.

 

2) First, select the target layer for which you selected a single feature.  You want to verify the estimates and margin of error variables, and you can adjust the confidence level from the default 90%.  Select OK.

 

3) The output is represented by four different symbologies.  First, your chosen county is filled with dots.  All counties that are significantly different are striped, while all those that are not are empty.  Finally, when significance cannot be determined, the original color fill is replaced with a new color.  In this case, median household income is not significantly different between Durham and Chatham counties.  However, this could be due to small differences or large margins of error in one or both counties.

 

 

GIS

ArcGIS Tutorial – Georeferencing Imagery

4 Comments

One of the limitations of computer mapping technology is that it is new. There is little historical imagery and data available as a result, although this has started to change. The integration of paper and imaged maps into computer mapping technology is possible, and this tutorial will walk through the process of georeferencing.

Georeferencing is the process of placing an image into two dimensional space. In essence, georeferencing pins a scanned map to particular geographical coordinates.

This tutorial will georeference a map of Durham County from 1955. In addition to the scanned map, we will use two current layers as referents: the Durham roads layer, and the Durham county boundary. Note that because the layers are more recent than the historical map, many roads will not exist in the image. Georeferencing historical imagery requires familiarity with geographic characteristics and changes.

 

Step 1: Enable Georeferencing

First, under the “Customize” Menu Bar option, navigate to “Toolbar” and select Georeferencing. The figure to the right displays the Georeferencing toolbar.

 

Step 2: Add Data and Image Layers

Next, add the shapefiles that you will use as referents for the image.

Once this is done, add the image to be georeferenced.  Note that you will almost certainly not see that image, as it lacks spatial coordinates. However, the image will appear in the Table of Contents.

In this example, I have added Durham County (blue polygon) and the Durham roads layer (blue lines).

 

Step 3: Fitting the Image to the Layers

The next step will relocate the image to the center of your current window and will expand the image only to the point where the entire image is visible. In this case, Durham County is taller than it is wide, so vertical space will be maximized.

First, it is a good idea to zoom, if necessary, so that your current view roughly matches where the image will be place. In this case, zooming to the full extent of the Durham county boundary will accomplish this.

Second, under the Georeferencing toolbar, click “Georeferencing” and select “Fit to Display.” The image should be roughly aligned to the data layers, though if not, this is not problematic.

As you can see from the image to the right, there is some distance between the county boundaries of today (red lines) to the hand-drawn county boundaries located in the image (white lines).

 

Step 4: Adjusting the Map

ArcGIS georeferences images through the addition of control points. The control points tool (to the right) operates through two mouse clicks: the first mouse click selects a point on the image, and the second mouse click pins that point to a location within a data layer.

For example, in the image to the right, I have selected a major intersection that likely has not changed in the last 60 years. After my first click, where I’ve selected a point near the top of the intersection, a green crosshair is placed. As I move the mouse, ArcGIS will pin my current crosshair to a proximate layer, in this case, the Durham roads layer.

Once you click a second time, the map will move to conform to the new control points. Control points work in combination, so as you add new control points, your image will (ideally) match more closely to your referents.

There is a limit to how much each subsequent control point will improve fit as more points are added. Generally, it’s a good idea to zoom in to improve accuracy and to create control points across the extent of the image.

After about 15 control points, we can compare the image to the included shapefiles. As you can see, if we assume that major roads have not changed, the green lines correspond well to the image, while the county boundary does to a lesser extent.

 

Step 5: Statistics and Transformations

Before saving the results, it is also a good idea to evaluate the results. Open the Table of Points to see each of your control points and the root mean squared error of all control points.

The Root Mean Square error (RMS) provides a rough guide to how consistent your control points are to one another with reference to the map.  Note that a low value does not mean that you’ve necessarily georeferenced the image well, it means you’ve georeferenced consistently.  High RMS errors indicate that your control points are less consistent with one another in comparison with a low RMS error.  One way to address this issue is to identify especially probelmatic control points and either replace or remove these points.  However, always reevaluate how well your image maps to the referent shapefiles.

You may delete control points or add new points at this stage. In addition, you may also try different transformations, although second- or third-order transformations are rarely needed.

 

Step 6: Saving the Results

Under the Georeferencing tab of the Georeferencing toolbar, select “Update Georeferencing.” Spatial information is saved in two new files that MUST accompany the image, an “.aux” file and a “.thw” file.

 

General Tips

– Zoom close to the layer resolution in order to improve accuracy

– Use more than 1 referent if possible. In this example, the county boundary provided a rough guide with respect to how far off the image initially is, but was not used to actually georeference the image.

– Georeference to accurate features. In this example, the county boundary was hand-drawn on the image and is not as precise as photographed features, like roads.

Data Visualization, GIS, spatial humanities

What’s new in ArcGIS 10?

Basemaps

Would you like to add aerial photography or a topographic map underneath map layers for visual appeal or context? With ArcGIS 10, you can add a basemap to your map project.

A basemap is a link to an online imagery data source. You must be connected to the Internet in order to see a basemap.

Basemaps contain imagery at different levels of detail. When zooming in or out, new imagery will replace old imagery, which provides an approprate level of detail at any zoom level and improves performance by limiting the amount of information to be downloaded and displayed.

 

Export Map Packages

Sharing maps and shapefiles with others can be a pain when a map is composed of many shapefiles and layers.  A map package bundles all shapefiles, layers, and map documents into a single file that can be opened by others with ArcGIS 10.

 

Background Processing

In ArcGIS 10, ArcToolbox tools default to background processing.  This allows you to continue to work while the tool processes your data.

To disable background processing, navigate to the “Geoprocessing Options…” choice under the Geoprocessing Menu Bar, and uncheck the “Enable” box.

 

Search Toolbox Feature

Got a tool you want to use but can’t remember what toolbox its in?  With the Search feature, you can easily locate what you need. Your search term can be the tool name or a close approximation of what you wish to do.

 

Easy to Use Time Data

Time series data became easier to use with ArcGIS 10. Version 10 recognizes time series data with the addition of a single time field.

For example, suppose you have annual precipitation for US cities.  Your data will contain an ID field, a point field, a time field containing the year, and a field containing the precipitation amount.

For more information, see this blog post.

 

How Do I Label Individual Items?

Have you ever wanted to label individudal items on a map, and avoid the cluttered appearance of labels for all features, such as that shown to the right?

ArcGIS 10 hides the tool that you use to label individual items, but it’s easy to get back.

  1. Turn on the “Labeling” toolbar under the Customize Menu Bar.
  2. At the top right corner of the toolbar, click the arrow pointed downward and click “Customize…”
  3. Select the “Commands” tab and select the “Label” category (left panel).
  4. In the right panel, drag the “Label” tool and drop it into any toolbar that you wish.
Uncategorized

Converting ArcGIS Layers to Google Earth (KML)

4 Comments

Converting ArcGIS layers to Google Earth allows others to easily see layers without specialized software.  Both ArcGIS and Google Earth Pro contain tools that allow conversion to and saving in KML format.
Note: Be certain you are allowed to share layers if they were not created by you.

Conversion using ArcGIS

  • First, open the layer that you wish to covert.
  • In the ArcToolbox window, expand “Conversion Tools,” then “To KML,” and select “Layer to KML.”
  • When the “Layer to KML” window appears, first select the shapefile or layer for the “Layer” box.
  • Next select a directory for the file to be created and provide a name for the file.
  • Finally, you must enter a number for the “Layer Output Scale.”  If your layer has a scale-dependent renderer, this setting allows you to export the KML at a specific level of resolution.  Otherwise, it has no effect, whatever the number.

For layers with many features, ArcGIS may produce a KML file that does not open in Google Earth due to errors.  There are two ways to solve this problem.

  • First, you can split your shapefile into several smaller shaepfiles.
  • Second, you can (usually) convert the shapefile to KML with Google Earth Pro.

Conversion using Google Earth Pro

  • First, open the shapefile with the Open command.  Be certain to change the file type to “ESRI Shapefile”.
  • When opened, you will receive a warning if your shapefile contains more than 2,500 items.  You will still possess the ability to import the entire file, but it may take some time.
  • You will be asked whether you wish to apply a style template to the document.  If you do so, you will be able to choose the attribute that contains the item name (for example, the address field or the street name field).
    Note: you don’t have to save the style template to select the name field.
  • Finally, right-click the layer added to the Temporary Places folder, and click “Save Place As.”  Provide a location and file name for the file to be created.
Data Visualization, economics, GIS

Time Series Visualizations in ArcGIS – An Introduction

9 Comments

Introduction

ArcGIS 10 makes it easy to manage and visualize time-series data to identify trends and create compelling visualizations.  Creating a visualization of time-series data requires only a few additional steps beyond those needed to produce any map.

Step 1: Data Formatting

Time-series data contains records, each of which is specific to both an individual and to a single point in time.  The following example uses employment data for the textile industry in North Carolina from 2000 through 2009.

In this example, “fips” corresponds to each county’s unique FIPS code, “industry” corresponds to the textile industry’s unique NAICS code representation, “t” denotes the year.  Establishments, employment, and annual pay, our data items, are stored in the fields “est”, “emp”, and “pay_ann”.  All missing values were coded ‘-1’.

Tip: Make sure each record has a value.  Records without values will not be drawn in ArcGIS.

Tip: Do not name the time field “year,” as it is a reserved name in ArcGIS.

We suggest based on experience that the storage of data in a Microsoft Access database provides the greatest degree of reliability.

Step 2: Add Data to Map in ArcGIS

Once the data is formatted, join the data to a geographic layer.  For help in finding a geographic layer, please consult the Perkins Data and GIS Services Department.

Tip: When joining layers, it is good practice to Verify the join selection before approving.  The program will inform you of any errors.

Step 3: Enabling Time

Once the data are joined to a layer, enter the layer properties by right-clicking the layer name in the Table of Contents pane.

Navigate to the Time tab and check the box.  ArcGIS will want to know which field contains time information, as well as the format.  If the join was successful, you will see the fields that represent the data joined to the geographic layer.  In this example, the time field is labeled “t”.

You must also specify the date/time format.  Available time formats are listed to the right.

Finally, you will have to enable time on the data table as well.  To do this, right-click the data table in the Table of Contents pane.  Follow the same steps as presented for the geographic layer.

Step 4: Enable Time Display

Now that ArcGIS understands the data structure, you may enable time visualization.  The “Tools” toolbar, which contains the most commonly used tools, contains the button highlighted below, “Open Time Slider Window”.  Select this button.

The time slider window (left) will appear.  The slider spans the time range of the data, identifies what point in this range is currently displayed on the map, and allows for access to a variety of playback and recording options.  To access these options, click the options button.

This button is the equivalent of “Play.”  It will display the data from the first time point to the last.

Buttons with both arrows and vertical lines are one-step increments.  This particular button moves forward one time increment, the other one moves back.

This button exports the display to video.  This is the final step.

Step 5: Configure Options and Visual Display

Before you export to video, you will want to configure the appearance of the map.  This example will focus on new options that come with time series data.

First, select “Options” in the Time Slider toolbar.  Under the “Time Display” tab, you can alter the format of the displayed date to conform to your data.  In this example, I selected 2011 (yyyy) because we are using annual data.

Second, under the “Playback” tab, you can specify a length of time for playback.  This example contains 10 years of data.  If I specify 5 seconds playback, each data year will be displayed for one-half second.  If I specify 10 second, each year will be visible for 1 second.

Third, I will display the year in order to make clear to the viewer the time point that is visible.  To do this, I will go to “Insert” “Dynamic Text” “Data Frame Time.”

Tip: Alternatively, you can insert the data frame time into the title or other display object by including the following in the text of the object: <dyn type=”dataFrame” name=”Layers” property=”time” emptyStr=”[off]”/>

After some trial and error, I successfully integrated the time currently visible into the title.  The image to the left shows its appearance.

Step 6: Export to Video

Once the appearance of the map is satisfactory, you can export the map to video or to sequential images.  Click the “Export to Video” button on the time slider window.

Tip: maximize the ArcGIS window, switch to Layout View, zoom the layout to 100%, and clear any toolbars that may obstruct the layout view to improve video appearance.

First, you will be asked for a file or folder location and the export format.  Videos are exported as AVI files, while sequential images are exported to a folder either as bitmaps or JPEGS.

Second, if you exported to video, you will be asked to select a codec, which essentially encodes and compresses the outputted video.  The codec selection depends on the individual machine, and some codecs work with ArcGIS better than others.

Finally, you may have to produce a video several times before it comes out as expected.  Be sure to watch for missing time points, as this frequently happens.  Fixing the video length to a specific play duration per time point (one-half second or one second) helps you watch for these missing time points.

The following example is a 5-second video that displays employment in the textiles industry in North Carolina from 2000 through 2009.  Note that declining employment is signified by colors that change from dark to light.