ModelBuilder

Ever have trouble conceptualizing your project workflow?  ModelBuilder  allows you to plan your project before you run any tools.  When using ModelBuilder in ESRI’s ArcMap, you create a workflow of your project by adding the data and tools you need.  To open ModelBuilder, click the ModelBuilder icon     (MB_Icon) in the Standard Toolbar.

MBIcon

Key Points Before You Build Your Model

ModelBuilder can only be created and saved in a toolbox.  In order to create your model, you first need to create a new toolbox in the Toolboxes, MyToolboxes folders in ArcCatalog.  Once you have a new toolbox, you will need to create a new Model; to do this, right click your newly created toolbox and select New, then Model.  When you wish to open an existing ModelBuilder, find your toolbox, right click your Model and select Edit.

In order to find the results of your model and the data created in the middle of your project workflow (also known as intermediate data), you will need to direct the data to any workspace or a Scratch Geodatabase.  To set your data results to a Scratch Geodatabase in ModelBuilder, click Model, then Model Properties.  A dialog box will open and you will want to select the Environments tab, Workspace category, and check Scratch Workspace.  Before closing the dialog box, select “Values” and navigate to your workspace or your geodatabase.

Set_Workspace

Building and Running a Model

To create a model, click the Add Data or Tool button (AddData).  Navigate to the SystemToolboxes, find the tool you wish to run, and add it to your model.  Double click the tool within the Model and its parameters will open.  Fill out the appropriate fields for the tool and select OK.

When the tools or variables are ready for processing, they will be colored blue, green, or yellow.  Blue variables are inputs, yellow variables are tools, and green variables are outputs.  When there is an error or the parameters have not been chosen, the variables will have no color.

ModelBlog_Good

Once you have your model built, click the Run icon (MBRun) to run the model.  Depending on the data and the amount of tools you run, the Model can take seconds or minutes to run.  You can also run one tool at a time; to do this, right click the tool and select “Run.”  When the Model is done running, the tools and outputs will have a gray background.  To find the results of your model, navigate to the Scratch Workspace you have set and add the shapefile or table to ArcMap or right-click the output variable before running the model and select “Add to Display.”

Applying ModelBuilder

The model above demonstrates how to take nationwide county data, North Carolina landmark data and North Carolina major roads data and find landmarks in Wake County that are within 1 mile of major roads.  The first tool in the model (Select Layer by Attribute tool) extracts Wake County from the nationwide counties polygon layer. 1

Once Wake County is extracted to a new layer, the North Carolina landmarks layer is clipped to the Wake County layer using the Clip tool2 The result of this tool creates a landmarks point layer in Wake County.  The third tool uses the Buffer tool on the primary roads layer in North Carolina.  Within the Buffer tool parameters, a distance of 1 mile is chosen and a new polygon layer is created.

 

Finally, the Wake County landmarks layer is intersected with the buffered major roads layer to create a final output using the Interect tool.4  Using ModelBuilder has many benefits: you document the steps you used to create your project and you can easily rerun the tool with different inputs after the model is built.  ModelBuilder allows users to easily determine if and where problems in the workflow are.  When there is an error in the workflow, a “Failed to Execute” message will appear and tell users which tool was unable to execute.  ModelBuilder also lets users easily change parameters.  In the model used above, you could change the Expression in the Select Layer by Attribute tool from ‘Wake’ to ‘Durham’ and find landmarks within 1 mile of major roads in Durham County.

ArcGIS Open Data

What is Open Data?

Finding data can be challenging.  Organizations and government agencies can share their data with the public using ESRI’s ArcGIS Open Data, a centralized spatial data clearinghouse.  Since its inception last year, over 1,600 organizations have provided more than 22,000 open datasets to the public.  Open Data allows users to find and download data in different formats, including shapefiles, spreadsheets, and KML documents, as well as APIs (GeoJSON or Esri GeoServices) to call the data into your own application.  It also lets you create various types of charts.

Search_Open_Data

How to Find and Use Data

Open Data allows consumers to type in a geographic area or a topic of interest in a single search box.  Once you’ve found data that appears to be what you were looking for, you can use the data for GIS purposes or use a table to create charts and graphs.  If you are looking for GIS data, you can preview the spatial data before downloading by clicking the “Open in ArcGIS” icon.  This takes users to ArcGIS Online where they can create choropleth maps and interact with the attribute table.   Users interested in tabular data can filter it and create various types of charts.  If more analysis of the data is necessary, you can download it by clicking the “Download Dataset” icon; you are able to download the entire dataset or the filtered dataset you’ve been working with.

OpenData_Page

Tips

The Source and Metadata links below the “About” heading provide information about the data.  In-depth information such as descriptions, attributes, OpenDataAboutand how the data was collected are provided in these links.  Below the name of the dataset there are three tabs:  “Details,” “Table,” and “Charts.”  Under the “Details” tab there are three sections, the Description, Dataset Attributes, and Related Datasets sections.  The Dataset Attributes section outlines the fields found within the dataset and provides field type information, while the Related Datasets section provides links to other datasets that have similar geographies or topics to the dataset you’ve chosen.  In the “Table” tab, you can view and filter the entire table in the dataset and the “Charts” tab allows you to create different charts.

OpenDataDetailTo obtain the most updated dataset or other updated articles related to the dataset, users should subscribe to the dataset they are interested in.  To subscribe, copy the link provided into an RSS Reader.  For specific data source questions, feel free to ask the Data and Visualization Department at askdata@duke.edu.

2015 Student Data Visualization Contest Winners

Our third year of the Duke Student Data Visualization Contest has come and gone, and we had another amazing group of submissions this year.  The 19 visualizations submitted covered a very broad range of subject matter and visualization styles. Especially notable this year was the increase in use of graphic design software like Illustrator, Photoshop, and Inkscape to customize the design of the submissions.  The winners and other submissions to the contest will soon be featured on the Duke Data Visualization Flickr Gallery.

As in the past, the submissions were judged on the basis of five criteria: insightfulness, broad appeal, aesthetics, technical merit, and novelty.  The three winning submissions this year exemplify all of these and tell rich stories about three very different types of research projects. The winners will be honored at a public reception on Friday, April 10, from 2:00 p.m. to 3:00 p.m, in the Brandaleone Lab for Data and Visualization Services (in the Edge).  They will each receive an Amazon gift card, and a poster version of the projects will be displayed in the lab.  We are very grateful to Duke University Libraries and the Sanford School of Public Policy for sponsoring this year’s contest.

First place:

Social Circles of Primary Caregivers / Tina Chen

Presentation2

Second place:

Crystal Structure of Human Proliferating Cell Nuclear Antigen (PCNA) for in silico Drug Screen / Yuqian Shi

DVC

Third place:

Deep and Extensive Impacts to Watershed Shape and Structure from Mountaintop Mining in West Virginia / Matthew Ross

DukeViz_MR

Please join us in celebrating the outstanding work of these students, as well as the closing of the Places & Spaces: Mapping Science exhibit, on April 10 in the Edge.

DataFest 2015 @ the Edge

DataFest 2015Duke Libraries are happy to host the American Statistical Association’s Data Fest Competition the weekend of March 20-22nd.  In its fourth year at Duke, DataFest brings teams of students from across the Research Triangle to compete in a weekend long competition that stresses data cleaning, analytics, and visualization skills.   The Edge provides a central location for the competition with facilities designed for collaborative, data driven research.

While the deadline for forming DataFest teams has past, Data and Visualization Services and Duke’s Department of Statistical Sciences are happy to offer another opportunity to participate in DataFest.  Starting Monday, March 16th we are offering four workshops on data analytics and visualization in the four days leading up to the DataFest event.  All workshops are open to the public, but we strongly encourage early registration to ensure a seat. Please come join us as we get ready to celebrate ASA DataFest 2015.

DataFest Workshop Series

Monday, March 16th, 6:00-8:00 PM – Introduction to R

Tuesday, March 17th, 1:30-3:00 PM – Easy Interactive Charts and Maps with Tableau

Wednesday, March 18th,  6:00-8:00 PM – Data Munging with R and dplyr

Thursday, March 19th, 7:00-9:00 PM – Visualization in d3

 

Sharing Files: Your Duke Box.com

Last fall Duke University released its newest file sharing service known as Duke’s Box.  By partnering with Box.comBox.com Logo, Duke offers a cloud-storage service which is intuitive, secure, and easy to use. Login with with your NetID, share files with colleagues, and have confidence this cloud storage is compliant with all laws and regulations regarding data privacy and security.

Simple to Use

Duke’s Box is similar to other cloud-based file storage services which support collaboration, productivity, and synchronization.  You can drop and drag files, identify collaborators and set permissions (read, edit, comment, etc.) But unlike some services, such as Dropbox or Google Drive, Duke’s Box enables you to be in compliance with data privacy and security. Additionally, you can synchronize data across your devices, at your discretion and subject to Duke’s Security & Usage Practice restrictions

While you may have previously used OIT’s NAS (Network Attached Storage) file storage service known as CIFS for data storage,  Duke’s Box is easier to use -although it provides services for slightly different use-cases. For example, CIFS might be more useful if accessing large files (e.g. video files that are larger than 5 GB). However, CIFS doesn’t enable collaboration or sharing.  Depending on your needs you may still want to use your departmental or OIT NAS.  Either way, you can use both file storage services and each service is free.

Check out this 5 minute quick-start video:

50 GB of Space by Default

You are automatically provisioned 50 GB of space, but you can request more if you need more.  See the FAQ for details.

Individual file size limitations are throttled to less than 5 GB.  This means Duke’s Box may be less than ideal for sharing very large files. NAS services may be more appropriate for large files as the time to download or synchronize large files can become inconvenient.  But for many common file sharing cases, Duke’s Box is ideal, fast and convenient.

Documentation, Restrictions & Use

While you can store many types of files, there are best practices and restrictions you will want to review.  For example, Duke Medicine users are required to complete an online training module prior to account activation.

  • Security and Use, including more detail on Terms of Service, and example Data Types — including military and space data,  FERPA, HIPAA, etc.
  • Duke’s Box Usage Practices
  • OIT’s FAQ
  • Your Duke’s Box “Read Me” folder. duke box - readmeOIT has done a great job of providing quick and convenient documentation located right where you need it.  See the READ ME folder after you logon to Duke’s Box.

Sharing Your Data With Us

One of the many use-cases for Duke’s Box is a more convenient way for you to share your data with us.  As you know we welcome questions about data analysis and visualization. We know describing data can be difficult while sharing your dataset can clarify your question.   But sharing your data via email consumes a lot of resources — both yours and ours. Now there’s a better way; please share your data with us via Duke’s Box.

Steps for Sharing Your Data with DVS Consultants

How to Share your files - 5 second annimated loop

  1. Log into Duke’s Box  (Use the bluecontinuebutton) 
  2. Open your “homefolder
  3. Put your data in the “sharingfolder
  4. Use the “invite people” button (right-hand sidebar)
    • Using a consultant email address, invite the DVS Consultant to see your data.  (Don’t worry if you don’t have our email yet.  When you start your question at askData@duke.edu, an individual consultant will be back in touch.)

Visualization Exhibit and Events

2015-01-07 16.32.31

ps_logoThis semester, Duke is proud to host the Places & Spaces: Mapping Science exhibit, visiting from Indiana University.  Places & Spaces is a 10-year effort by Dr. Katy Börner (director of the Cyberinfrastructure for Network Science Center) to bring focus to visualization as a medium of scholarly communication.

20150105_105415The exhibit includes 100 maps from various disciplines and cultures and highlights myriad visualization techniques that have been used to communicate science to a broader public. The maps are divided among three spaces on campus: The Edge (newly opened on the first floor of Bostock Library), Smith Warehouse (on the second floor of Bay 11), and Gross Hall (on the third floor).

KatyBorner_weblrgTo celebrate the opening, Dr. Börner will visit Duke on January 21st and 22nd.  She will give a keynote presentation on Wednesday, January 21, at 4pm, in the Edge.  A reception will follow.

Additional events next week and throughout the semester will celebrate the exhibit and promote ongoing visualization work at Duke.  All events are open to the public!

Upcoming events

Wednesday, January 21

Thursday, January 22

Friday, January 23

More information about the exhibit and related events is available at:
http://sites.duke.edu/scimaps/ and
http://scimaps.org/duke

Please contact Angela Zoss (angela.zoss@duke.edu) with any questions or suggestions.  We hope you can join us in celebrating and enjoying this exhibit!

New Year- New Data and Visualization Lab!

Data and Visualization Services is happy to announce our new Data and Visualization Lab in Duke Libraries new Edge research space.  Located on the first floor of the Bostock Library, the Brandaleone Family Lab for Data and Visualization Services offers a dedicated space for researchers working on data driven projects.

The lab features three distinct areas for supporting data driven research.

Data and Visualization Lab Space

Data and Visualization Lab Computing Zone

Our lab space features twelve high end workstations with dual monitors with the latest software for data visualization, digital mapping, statistics, and qualitative research.  All of the machines have two dedicated displays to encourage collaborative work and data consultations.  Additionally, all twelve machines have a dedicated power port located conveniently under the edge of the table for powering a laptop or usb powered device.

Bloomberg Professional “Bar”

bloom

Since the launch of our Bloomberg terminals, we have seen a steady increase in both individual and team based usage of Bloomberg financial data.  Our three Bloomberg Professional workstations are now located on a dedicated “bar” across from our lab machines.  The  new Bloomberg zone will facilitate collaborate work and provide a base for groups such as the Duke University Investment Club and Duke Financial Economics Center.

Consult and Collaborative SpaceCollaboration Zone

Our third lab space provides a set of four rolling tables for small groups to collaborate or for projects that don’t require a fixed computing space.   An 85″ flat panel display near this zone features data visualizations and other data driven research projects at Duke.

Come See Us!

With ample natural light,  almost 24/7 availability, and a welcoming staff eager to work with you on your next data driven project.  We look forward to working with you in the upcoming year!

Enter the 2015 Student Data Visualization Contest

contest_blog-01Calling all Duke undergrad and grad students! Have you worked on a course or research project that included some kind of visualization? Maybe you made a map for a history class paper. Maybe you invented a new type of chart to summarize the results of your experiment. Maybe you played around with an infographic builder just for fun.

Now is the time to start thinking about submitting those visualizations to the Duke Student Data Visualization Contest. It’s easy — just grab a screenshot or export an image of your visualization, write up a short description explaining how you made it, and submit it using our Sakai project site (search for “2015 DataVis Contest”). The deadline is right after finals this fall, so just block in a little extra time at the end of the semester once you’re done with your final assignments and projects.

Not sure what would work as a good submission? Check out our Flickr gallery with examples from the past two years.

Not sure if you’re eligible? If were a Duke student (that is, enrolled in a degree granting program, so no post-docs) any time during 2014, and you did the work while you were a student, you’re golden!

Want to know more about the technical details and submission instructions? Check out the full contest instruction site.

Story Maps

Telling Stories with Maps

StoryMap Pic1“Story maps” are a popular method of telling place-themed stories and engaging with your audience over the web. Story maps are highly interactive, allowing users to follow along a path or time-line with links to content along the way. They’re also a great way to visualize current events and news topics in a way that brings perspective and context to important issues. As a student or researcher, you can use maps to tell a story about your research study area. In that sense, they can be a great tool for drawing attention to your work, and you could consider it another form of social media.

Creating a web map may seem like a challenge if you’ve never done it before, but there are several tools available online that can quickly and easily generate a story map. For this post, I’ll introduce you to two different types of story maps and suggest some free tools for creating your own.

Mapping Places or Events 

Story maps that cover a series of events are useful for contextualizing news events, giving an online tour, or linking to almost any kind of location specific information. Story maps of this style are fun to use because they typically provide both a map and multimedia content. The user accesses the information in an interactive format -which is a great way for your message to sink in!

For example, I created this story map that links historic building photos of the Construction of Duke University to their locations on a map.

View the full size map here.

Some applications for this type of story map are publishing information about research areas, adding new points of access for digital humanities, or documenting travel or a field expedition.

Thematic Maps

Another popular style of story map is one that presents a series of thematic maps. These types of maps often depict how changes have occurred over time in a place or perhaps the unfolding of a news event. Side-by-side comparison of maps can also be a visually interesting way to illustrate an important issue. An interesting comparison map might show US Census demographic data from different census years in a city to show how people have changed.

This map illustrates how manufacturing jobs have changed around Flint Michigan from 1990 to 2010.

Click here for full size.

There are also some really cool interactive features out there for this style of map like a tabbed viewer, a swipe or slide function between two different maps, and ESRI’s SpyGlass.

Create Your Own!

Some great tools are available to the Duke community and freely on the web that let you create these types of “story maps” with minimal training. Here are three tools you can use and what each does best…

StoryMap JS is a completely free and open access tool by the Knight Lab at Northwestern University. A Google account is necessary because StoryMap JS actually saves the maps you create in the recent folder of your Google Drive. StoryMap JS is incredibly easy to use, too. It has a very simple and intuitive interface that will let you start making your map in minutes. You can also use StoryMap JS for non-cartographic visual materials, and there is a cool off-shoot that allows your to instantly map 20 recent geo-tagged Instagram photos from any user account. Best Uses: Try using StoryMap JS when you’re telling a story that unfolds over a path or timeline. It’s also great for linking to media like photos or YouTube content.

Social Explorer You may have used Social Explorer before to gather US Census data, but you can also create thematic maps that you can share or embed in a website. With your Duke credentials, you have access to the Professional Edition. The data is pre-loaded, so you’re just a few clicks away from a beautifully shaded thematic map of US Census Data that you can share over the web. The map interface is user-friendly and has a “Change Layout” button at the bottom center that creates side-by-side and swipe comparison maps. You can also create an annotated presentation that let’s the user cycle through a series of maps. Here is a quick example of a map presentation I made in Social Explorer. Best Uses: Social Explorer’s best use is for mapping US Census data. The “Tell a Story” function allows you to join graphs and other media to your map and create interactive presentation slides.

ESRI ArcGIS Online For more advanced users, or just those looking for more customization options, ArcGIS Online offers an abundance of tools and templates for creating attractive and engaging map presentations. ArcGIS Online Story Maps require an account with ESRI. You can sign up for a free public account, or, for more advanced features, you can request a free organizational account that is available to the Duke community. To take advantage of all ArcGIS Online has to offer, you will need to familiarize yourself with the how to use it. Once you’ve made a few maps, you can load maps and multimedia content into any of ESRI’s Story Map Apps. Take a look at this gallery of to view some examples of what you can do with Story Maps in ArcGIS Online. Even though there is a bit of a learning curve to ArcGIS Online, the pay off is huge.

Here is a customized slider map I made using the Story Map Swipe App that shows changes in North Carolina’s Congressional District Boundaries following the 2012 redistricting. Use the slider to swipe between views.

View the full size map here.

Best Uses: Fully customized story maps of any type. Great for telling place-based stories and presenting a series of thematic maps complete with multimedia content.

I hope you enjoyed viewing some of these story maps! I’m sure you can see that there are many different uses for this type of media. If you’ve made a cool story map, feel free to share it with us in the comments!

Welcome to the Current Population Statistics on the Web

Duke University recently acquired access to the online version of its Current Population Statistics (CPS) CD-ROM collection to facilitate easy access to CPS data (Unicon’s CPS Utilities on the Web).  This blog post will walk through the basic data extraction process.  The interface is comparable to that provided by the CD, and users of this collection will find the interface and powerful.  Please note that the instructions provided on the web site are very important to read, particularly for those unfamiliar with the CPS CD version.

Create an Account

When you visit the Unicon site (http://unicon.com/), click the “CPS on Web” link to the left, then click the Register button.  You will have to enter some information to complete the registration process.

Once complete, submit the information.  Once the registration window closes, choose the CPS series (or month) you wish to query, and log in to the system.

 

1Navigation and Data Extraction

Once logged in, you will see a popup window like that shown in the image to the right.  For a typical data extraction, the following steps are advised.

1) First, click the Set Option button and chang4e the timeout to at least 300 seconds.  This will ensure successful data extraction.

2) Next, click the Make an Extraction button, followed by the Request Editor button on the next page.  You should see a page similar to that below (all variables used in your prior extraction will be listed).

23) Remove any variables you do not need.  Next, make certain the variable you wish to include is selected at the top and click “Add Variable(s).”  Alternatively, if you already know the names of the variables, you may type them into the boxes provided on the page.

4) Once all variables are added to the selection, click Continue.  On the following page, specify the output format for the dataset.  Once complete, be certain to select one or more years (at the top).  After you have selected years, click the Extract button.

5) On the following page, you will be presented with a list of variables by year.  As variables change across years in some cases, not all selected variables may be present for each year.  When selecting variables, checking the “View Documentation” checkbox at the top will allow for browsing of available years.

 

Other Useful Tools

– The Make a Table button allows for the construction of crosstabs of observations, means, and other statistics.  This is helpful if the goal is to locate variables for analysis or if there is a choice between two or more variables.

– The Make a Graph button is also useful for data exploration.  The program provides the ability to construct hsitograms, line charts, scatter ploys, pie charts, and bar charts.  Basic summaries of a variable can also be generated from this page.

– If your data need to be weighted to represent the US population, be certain to select the appropriate weight under the Apply Weights button before extraction.

– Subsets of individuals can also be produced under the Specify Universe button.  For example, a specific race or gender can be specified to reduce the sample to what you need.