Category Archives: Uncategorized

Data Curation, Data Management, Data Visualization, GIS, rstats, Statistics, Uncategorized

Data and GIS Fall 2013 Newsletter

1 Comment

Analyze, discover, manage, map, and visualize your data with Duke Libraries Data and GIS Services.  Our team of five consultants provides a broad range of support in areas ranging from data analysis, data visualization, geographic information systems, financial data, statistical software and data storage and management.  Our lab provides 12 workstations with the latest data software and three Bloomberg Professional workstations nearly 24/7 for the Duke community.

Data and GIS Workshop Series

All are welcome to the Data and GIS Workshop Series.  Analyze, communicate, clean, map, represent and visualize your data with a wide range of workshops on data based research methods and tools.  Details and registration for each class are available at the links that follow.  (Interested in keeping up to date with workshops and events in Data and GIS?  Just go to https://lists.duke.edu/sympa/info/dgs-announce and click on the “Subscribe” link at the bottom left.)

    Tue, Sep 3, 2013      1:00 PM - 3:00 PM    Introduction to ArcGIS    
    Wed, Sep 4, 2013     10:00 AM - 11:30 AM   Stata for Research    
    Wed, Sep 11, 2013    10:00 AM - 11:00 AM   Open (aka Google) Refine     
    Thu, Sep 12, 2013     1:00 PM - 3:00 PM    Analysis with R    
    Tue, Sep 17, 2013     1:00 PM - 2:30 PM    Introduction to Tableau Public 8    
    Thu, Sep 19, 2013    10:00 AM - 11:00 AM   Google Fusion Tables    
    Mon, Sep 23, 2013     1:00 PM - 2:30 PM    Introduction to Tableau Public 8    
    Tue, Sep 24, 2013     1:00 PM - 2:30 PM    Stata for Research    
    Mon, Sep 30, 2013    10:00 AM - 11:00 AM   Top 10 Dos and Don'ts for Charts and Graphs    
    Mon, Sep 30, 2013     1:00 PM - 3:00 PM    Introduction to ArcGIS    
    Tue, Oct 8, 2013      1:00 PM - 2:30 PM    Introduction to Text Analysis    
    Thu, Oct 10, 2013     1:00 PM - 3:00 PM    ArcGIS Special Topics: Geocoding & Proximity Analysis    
    Thu, Oct 17, 2013     1:00 PM - 3:00 PM    Historical GIS    
    Mon, Oct 28, 2013     1:00 PM - 2:00 PM    Designing Academic Figures and Posters    
    Tue, Oct 29, 2013     1:00 PM - 3:00 PM    Web GIS Applications

Data and GIS also offers instruction tailored to courses or research teams. Please contact askdata@duke.edu to schedule a session!

Data Management

Data Management Planning – DMPTool – Get 24/7 online help for your next data management plan, including information about Duke resources available for your data work.

Statistical Software Updates

Explore all of our Data and GIS Lab resources on our site at http://library.duke.edu/data/about/lab.html or come visit us on the second floor of Perkins Library.

Job Opportunities in Data and GIS Services

Data & GIS Services is hiring!  We have two open positions for student web programmers interested in working on data visualization projects.  See the Library Student employment page (http://library.duke.edu/jobs/students.html) for more information on how to apply.  (The job can be found by searching for requisition number “DUL14-AMZ02”.)

New Data and Map Collections

CPS on Web (CPS Utilities Online)
CPS on Web is a set of utilities enabling you to access CPS data and documentation from this website.   You may make tables and graphs from the CPS data, download data extractions, make estimations, get summaries and statistical measures, search the documentation, and make your own variables as functions of the existing ones.

Global Financial Data
Global Financial Data is a collection of financial and economic data provided in ASCII or Excel format. Data includes: long-term historical indices on stock markets; Total Return data on stocks, bonds, and bills; interest rates; exchange rates; inflation rates; bond indices; commodity indices and prices; consumer price indices; gross domestic product; individual stocks; sector indices; treasury bill yields; wholesale price indices; and unemployment rates covering over 200 countries.

LandScan Global
The LandScan Global Population Database provides global population distribution in a gridded GIS format at 30 arc-second resolution (approximately 1×1 km cells). Oak Ridge National Laboratory developed modeling techniques to disaggregate and interpolate census data within administrative boundaries to create a GIS layer showing population distribution as accurately and as timely as possible. EastView provides this data to use in GIS software as a WMS (Web Mapping Service) or as a WCS (Web Coverage Service) to allow a user to incorporate population distribution into GIS mapping and analysis.

Contact Us

email: askdata@duke.edu
twitter: duke_data or duke_vis

Uncategorized

Data and GIS Spring Semester News

New workshops for Spring 2013
http://library.duke.edu/data/news/index.html

Clean your data with Google Refine.  Use digital maps to explore the present and past.  Analyze data with R or Stata. Visualize your research with one of our data visualization courses.  The Data and GIS Workshops offer a range of research strategies for data based questions. Register online for our courses or schedule a session for your course by emailing askdata@duke.edu

Visualize This (and win a $500 technology prize)!
http://blogs.library.duke.edu/data/2012/12/04/2013-data-visualization-contest/

Are you a current Duke University undergraduate or graduate student? Have you used data visualization in a past or current research project to help solve a problem, tell a story, or highlight an interesting trend? Write up a short description and you’ll have a submission for the contest and a chance to win a $500 technology prize.

New Data Lab
http://library.duke.edu/data/about/lab.html

As mentioned in the fall – with 12 workstations with dual 24″ monitors and 16 gigs of memory, the new Data and GIS lab is ready to take on the most challenging statistical, mapping, and visualization research projects. The new lab also features a flatbed scanner for projects moving from print to digital data. Lab hours are the same hours as Perkins Library (almost 24/7).

Get help with Data Management Planning
http://library.duke.edu/data/guides/data-management/index.html

Puzzled by data management planning?  Not sure what to include in your grants data management plan?  Data and GIS has launched a guide that supports researchers looking for advice on data management plans now required by several granting agencies.  The guide provides examples of sample plans, key concepts involved in writing a plan, and contact information for groups on campus providing data management advice.

Get Data Help
http://library.duke.edu/data/about/staff.html

Come visit us in Perkins 226 for a consultation or contact us online (email: askdata@duke.edu or twitter: duke_data OR duke_vis).  Our consultants are available weekdays 8-5 by appointment or offer drop in hours as well We look forward to working with you on your next data driven project.

 

Data Visualization, GIS, Uncategorized

Online Mapping Tools – GeoCommons

Visualizing spatial data can be challenging.  Specialized software tools like ArcGIS produce excellent results, but often seem complex for relatively simple tasks. Several online tools have emerged recently that provide relatively easy alternatives for the display of spatial data.  In this post, we examine GeoCommons, a web based tool for presenting spatial data in detail.  (Go to this guide to see a comparison chart of packages and features, and see this Duke University Libguide for a more detailed review of GeoCommons.)

 

GeoCommons (geocommons.com)

GeoCommons is an online mapping application that easily imports a variety of data formats, including geospatial data, and quickly produces sharable maps.  In contrast to other mapping tools, GeoCommons contains several categorization algorithms, such as quantile classification and classification based on the standard deviation of the sample that assist with the construction of informative maps.  CSV files and ArcGIS shapefiles are two of the most widely used file formats compatible with GeoCommons.

GeoCommons is very easy to use and contains some of the display features contained in high-end GIS suites.  Creation of new variables tied to geographies can be tricky, so it’s advised to either upload data  and map in final form or to first identify the layer to which you will upload and join a complete data set.

 

Geocoding

Figure 1

To begin geocoding, upload a file.  Gecommons has the capability to recognize spatially encoded data.  Some formats may require user assistance.

If you’ve uploaded data that contains latitude and longitude coordinates, choose this option.  In my case, I had county FIPS codes that uniquely identified each county.  Selecting US Boundaries to the left, then USA Counties, I was able to successfully preview how well my FIPS codes matched the layer (Figures 1 and 2).  A variety of other boundary types are available.  The key is to have in your data a unique identifier that identifies each record in the same manner as an available geocoding layer.

Figure 2

Review the geocoding results and select Continue to proceed.

 

Mapping

Geocommons offers some nice built in features that assist with categorizing measures.  The application will produce summary statistics for numeric fields (Figure 3), which gives you a quick picture of your sample and can assist with how to categorize the data.  Click the “Make a Map” button to proceed to the interactive interface.

Figure 3

Also note the filter tab, which allows you to screen out groups of cases.  For example, I may request a minimum number of farms to screen out urban counties.

Figure 4 shows a standard choropleth map portraying median number of acres per farm by county for North Carolina in 2007.  In this example, I have classified counties into five groups using standard deviations to group counties.

 

Sharing

Figure 4

GeoCommons contains a wide variety of ways to share data (accessed through the About section).  Posting to Twitter, Facebook, and an array of other social media sites is possible with a few short clicks.  You can directly email a link to the map along with a short personal message right out of the application.For those who wish to post to a web page, GeoCommons provides two ways to insert a map, through a <div> tag and through an iframe.  All code is generated for copy and paste into your page.

To access a version of this map, simply follow this link.

Finally, GeoCommons will produce a PNG image and a KML document for download.  The image export feature appears to be relatively new and does take trial-and-error to align correctly.  In addition, it does not appear to include any base layers or legends in the output, only the data layer.

 

Other Notes

When using standard deviation and maximum breaks methods for grouping observations, double check the category definitions by changing the number of categories and the resulting changes to the definitions for the new groups.  This will help to confirm whether data are grouped appropriately and exactly what the definitions for each category are.

Uncategorized

Data and GIS Winter Newsletter 2012

Data driven teaching and research at Duke keeps growing and Perkins Data and GIS continues to increase support for researchers and classes employing data, GIS, and data visualization tools.  Whether your discipline is in the Humanities, Sciences, or Social Sciences, Perkins Data and GIS seeks to support researchers and students using numeric and geospatial data across the disciplines.

New Website for 2012
http://library.duke.edu/data/

You can find:

  • Online data or digital maps that you need for your project
  • A workshop on the latest software packages and digital tools

New workshops for 2012
http://library.duke.edu/data/news/index.html
Clean your data with Google Refine. Learn about data management planning. Visualize your data with Tableau Public, or map your results using ArcGIS or Google Earth Pro.  A new series of workshops connects traditional statistical, geospatial, and visualization tools with web based options.  Register online for our courses or schedule a session for your course by emailing askdata@duke.edu

  • StataReview                               (Statistics/Data Management)
  • Introduction to ArcGIS           (Geographic Information Systems / Data Visualization)
  • Data Management Planning  (Data Management/Grants)
  • Geocommons                            (Geographic Information Systems / Data Visualization)
  • Google Earth (Pro)                   (Geographic Information Systems / Data Visualization)
  • Google Refine                           (Data Management/Descriptive Statistics)
  • Tableau Public                          (Data Visualization)

Bloomberg (terminals) have arrived
http://blogs.library.duke.edu/data/2011/08/29/bloomberg-has-arrived/
Duke Libraries in pleased to announce the installation of three Bloomberg financial terminals in the Data and GIS Lab in 226 Perkins.  The terminals provide the latest news and financial data and include an application that makes it easy to export data to Excel.  Access is restricted to all current Duke affiliates.

Get help with Data Management Planning
http://library.duke.edu/data/guides/data-management/index.html
Data and GIS has launched a new guide that provides guidance for researchers looking for advice on data management plans now required by several granting agencies.  The guide provides examples of sample plans, key concepts involved in writing a plan, and contact information for groups on campus providing data management advice.

New Collections
http://library.duke.edu/data/collections/new.html
Explore the Indonesian Village Potential Statistics (PODES), look at household economic behavior in the Indian National Sample Survey, or explore historical digital maps of Europe- the Data and GIS collection collects research data sets and maps of interest to the Duke community covering a wide range of topics.

Support for Restricted Data Contracts and Restricted Data Licensing
Perkins Library has partnered with the Social Science Research Institute (SSRI) to support restricted data licensing with Paul Pooley as a restricted data specialist.  Paul is available  to work with researchers licensing restricted data and negotiating restricted data management plans.  Please contact Paul paul.pooley@duke.edu or askdata@duke.edu for more details.

Contact Us! – askdata@duke.edu – twitter: duke_data – http://library.duke.edu/data/hours.html

Joel Herndon
Head, Data and GIS Services
919-660-5946
Location: Room 227 Perkins
joel.herndon@duke.edu		 Mark Thomas
Economics/GIS Librarian
919-660-5853
Location: Room 233 Perkins
mark.thomas@duke.edu		 Teddy Gray
Biological Sciences Librarian
919-660-5971
Location: Room 233 Perkins
teddy.gray@duke.edu
Uncategorized

Converting ArcGIS Layers to Google Earth (KML)

4 Comments

Converting ArcGIS layers to Google Earth allows others to easily see layers without specialized software.  Both ArcGIS and Google Earth Pro contain tools that allow conversion to and saving in KML format.
Note: Be certain you are allowed to share layers if they were not created by you.

Conversion using ArcGIS

  • First, open the layer that you wish to covert.
  • In the ArcToolbox window, expand “Conversion Tools,” then “To KML,” and select “Layer to KML.”
  • When the “Layer to KML” window appears, first select the shapefile or layer for the “Layer” box.
  • Next select a directory for the file to be created and provide a name for the file.
  • Finally, you must enter a number for the “Layer Output Scale.”  If your layer has a scale-dependent renderer, this setting allows you to export the KML at a specific level of resolution.  Otherwise, it has no effect, whatever the number.

For layers with many features, ArcGIS may produce a KML file that does not open in Google Earth due to errors.  There are two ways to solve this problem.

  • First, you can split your shapefile into several smaller shaepfiles.
  • Second, you can (usually) convert the shapefile to KML with Google Earth Pro.

Conversion using Google Earth Pro

  • First, open the shapefile with the Open command.  Be certain to change the file type to “ESRI Shapefile”.
  • When opened, you will receive a warning if your shapefile contains more than 2,500 items.  You will still possess the ability to import the entire file, but it may take some time.
  • You will be asked whether you wish to apply a style template to the document.  If you do so, you will be able to choose the attribute that contains the item name (for example, the address field or the street name field).
    Note: you don’t have to save the style template to select the name field.
  • Finally, right-click the layer added to the Temporary Places folder, and click “Save Place As.”  Provide a location and file name for the file to be created.
biology, Biostats, Uncategorized

What’s hot in molecular biology databases

1 Comment

The journal Nucleic Acids Research has just published its 18th annual database issue. The current issue summarizes 96 new and 83 previously reviewed molecular biology databases, including GenBank, ENA, DDBJ, and GEO. Also included in the issue is an editorial advocating the creation of a “community-defined, uniform, generic description of the core attributes of biological databases,” which would be known as the BioDBCore checklist. Such a checklist would benefit both database users and provides: users would have a much easier time finding the appropriate resource and providers would be able to highlight specialized resources and the lesser known functionality of established databases.

Besides the databases reviewed in the current issue, Nucleic Acids Research maintains a select list of 1330 molecular biology databases that have been profiled in various database issues over the past 18 years.

Uncategorized

Welcome!

Welcome to the Perkins Data and GIS blog!  Our goal is to highlight Duke research, collections, policies, and tools surrounding empirical
data and digital maps of interest to the research community.  We hope that this blog will serve as a catalyst to link researchers and resources across the Duke community and beyond!