All posts by Joel Herndon, Ph.D.

Data Fest 2016 Workshop Series

2016-03-22 Joel Herndon, Ph.D. Leave a comment

Duke Libraries are happy to welcome the 2016 ASA DataFest to the Edge on April 1-3rd. As part of DataFest 2016, the Edge is hosting five DataFest related workshops designed to help teams and others interested in data driven research expand their skills. All workshops will meet in the Edge Workshop Room (1st Floor Bostock Library). Laptops are required for all workshops.

We wish all the teams success in the competition and hope to see you in the next few weeks!

DataFest Workshop Series

Data Analysis with Python
Tuesday, March 22
6:00-9:00 PM
This will be a hands-on class focused on performing data analysis with Python. We’ll help participants set-up their Jupyter Notebook development environment, cover the basic functions for reading and manipulating data, show examples of common statistical models and useful packages and show some of the python visualization tools.

Introduction to R
Wednesday March 23
6:00-8:00 PM
Introduction to R as a statistical programming language. This session will introduce the basics of R syntax, getting data into R, various data types and classes, etc. The session assumes no or little background in R.

Data Munging with R and dplyr
Monday, March 28
6:00-8:00 PM
This session will demonstrate tools for data manipulation and cleaning of data in R. Majority of the session will use the dplyr and tidyr packages. Some background in R is recommended. If you are not familiar with R, make sure to first attend the first R workshop in the series.

Data visualization with R, ggplot2, and shiny
Wednesday, March 30
6:00-8:00 PM
This session will demonstrate tools for static and interactive data visualization in R using ggplot2 and shiny packages. Some background in R is recommended. If you are not familiar with R, make sure to first attend the first R workshop in the series.

EDA and Interactive Predictive Modeling with JMP
Thursday, March 31
4:00-6:00 PM
JMP® Statistical Discovery Software is dynamic, visual and interactive desktop software for Windows and Mac. In this hands-on workshop we see tools for exploring, visualizing and preparing data in JMP. We’ll also learn how to fit a variety of predictive models, including multiple regression, logistic regression, classification and regression trees, and neural networks. A six month license of JMP will be provided.

Data Management, GIS, rstats, spatial humanities, stata, Statistics, workshops

Data and Visualization Spring 2016 Workshops

2016-01-11 Joel Herndon, Ph.D. Leave a comment

SPRING 2016: Data and Visualization Workshops

Interested in getting started in data driven research or exploring a new approach to working with research data? Data and Visualization Services’ spring workshop series features a range of courses designed to showcase the latest data tools and methods. Begin working with data in our Basic Data Cleaning/Analysis or the new Structuring Humanities Data workshop. Explore data visualization in the Making Data Visual class. Our wide range of workshops offers a variety of approaches for the meeting the challenges of 21st century data driven research. Please join us!

Workshop by Theme

DATA SOURCES

Structuring Humanities Data (Feb2) – NEW *

Web Scraping and Gathering Data from Websites (Mar2, Mar 10)

DATA CLEANING AND ANALYSIS

OpenRefine: Data/Text Cleaning, Mining and Transformations (Jan20, Feb16) *

Regular Expressions (Feb 18) – NEW *

DATA ANALYSIS

Basic Data Cleaning and Analysis for Data Tables (Jan 22, Feb 10) *

Advanced Excel for Data Projects (Feb 1, Feb 23)

Introduction to Stata (Feb 2)

Analysis with R (Feb 24)

MAPPING AND GIS

Introduction to ArcGIS (Jan 27, Feb 25)

Introduction to QGIS (Feb 3) – NEW

Historical GIS (Jan 28)

ArcGIS Online (Feb 15)

DATA VISUALIZATION

Easy Interactive Charts and Maps with Tableau (Jan 25, Feb 11)

Making Data Visual (Jan 29) NEW *

Advanced Tableau (Data Structures) (Feb 17) NEW

Adobe Illustrator for Diagrams and Visualizations (Feb 22) NEW *

Designing Academic Figures and Posters (Mar 4) *

* – For these workshops, no prior experience with data projects is necessary! These workshops are great introductions to basic data practices.

Data Analysis, Data Management, Data Sources, Data Visualization, GIS, rstats, spatial humanities, stata, Statistics, workshops

DVS Fall Workshops

2015-08-11 Joel Herndon, Ph.D. Leave a comment

Data and Visualization Services is happy to announce its Fall 2015 Workshop Series. With a range of workshops covering basic data skills to data visualization, we have a wide range of courses for different interests and skill levels.. New (and redesigned) workshops include:

OpenRefine: Data Mining and Transformations, Text Normalization
Historical GIS
Advanced Excel for Data Projects
Analysis with R
Webscraping and Gathering Data from Websites

Workshop descriptions and registration information are available at:

library.duke.edu/data/news

Workshop	Date
OpenRefine: Data Mining and Transformations, Text Normalization	Sep 9
Basic Data Cleaning and Analysis for Data Tables	Sep 15
Introduction to ArcGIS	Sep 16
Easy Interactive Charts and Maps with Tableau	Sep 18
Introduction to Stata	Sep 22
Historical GIS	Sep 23
Advanced Excel for Data Projects	Sep 28
Easy Interactive Charts and Maps with Tableau	Sep 29
Analysis with R	Sep 30
ArcGIS Online	Oct 1
Web Scraping and Gathering Data from Websites	Oct 2
Advanced Excel for Data Projects	Oct 6
Basic Data Cleaning and Analysis for Data Tables	Oct 7
Introduction to Stata	Oct 14
Introduction to ArcGIS	Oct 15
OpenRefine: Data Mining and Transformations, Text Normalization	Oct 20
Analysis with R	Oct 20

Data Analysis, Data Sources, Data Visualization

DataFest 2015 @ the Edge

2015-03-12 Joel Herndon, Ph.D. 2 Comments

Duke Libraries are happy to host the American Statistical Association’s Data Fest Competition the weekend of March 20-22nd. In its fourth year at Duke, DataFest brings teams of students from across the Research Triangle to compete in a weekend long competition that stresses data cleaning, analytics, and visualization skills. The Edge provides a central location for the competition with facilities designed for collaborative, data driven research.

While the deadline for forming DataFest teams has past, Data and Visualization Services and Duke’s Department of Statistical Sciences are happy to offer another opportunity to participate in DataFest. Starting Monday, March 16th we are offering four workshops on data analytics and visualization in the four days leading up to the DataFest event. All workshops are open to the public, but we strongly encourage early registration to ensure a seat. Please come join us as we get ready to celebrate ASA DataFest 2015.

DataFest Workshop Series

Monday, March 16th, 6:00-8:00 PM – Introduction to R

Tuesday, March 17th, 1:30-3:00 PM – Easy Interactive Charts and Maps with Tableau

Wednesday, March 18th, 6:00-8:00 PM – Data Munging with R and dplyr

Thursday, March 19th, 7:00-9:00 PM – Visualization in d3

big data, Data Curation, Data Management, Data Sources, Data Visualization, GIS, spatial humanities

New Year- New Data and Visualization Lab!

2015-01-07 Joel Herndon, Ph.D. Leave a comment

Data and Visualization Services is happy to announce our new Data and Visualization Lab in Duke Libraries new Edge research space. Located on the first floor of the Bostock Library, the Brandaleone Family Lab for Data and Visualization Services offers a dedicated space for researchers working on data driven projects.

The lab features three distinct areas for supporting data driven research.

Data and Visualization Lab Space

Our lab space features twelve high end workstations with dual monitors with the latest software for data visualization, digital mapping, statistics, and qualitative research. All of the machines have two dedicated displays to encourage collaborative work and data consultations. Additionally, all twelve machines have a dedicated power port located conveniently under the edge of the table for powering a laptop or usb powered device.

Bloomberg Professional “Bar”

Since the launch of our Bloomberg terminals, we have seen a steady increase in both individual and team based usage of Bloomberg financial data. Our three Bloomberg Professional workstations are now located on a dedicated “bar” across from our lab machines. The new Bloomberg zone will facilitate collaborate work and provide a base for groups such as the Duke University Investment Club and Duke Financial Economics Center.

Consult and Collaborative Space

Our third lab space provides a set of four rolling tables for small groups to collaborate or for projects that don’t require a fixed computing space. An 85″ flat panel display near this zone features data visualizations and other data driven research projects at Duke.

Come See Us!

With ample natural light, almost 24/7 availability, and a welcoming staff eager to work with you on your next data driven project. We look forward to working with you in the upcoming year!

big data, Data Analysis, Data Curation, Data Management, Data Visualization

Meet Data and Visualization Services

2014-08-18 Joel Herndon, Ph.D. Leave a comment

Data and Visualization Services Logo The fall of 2014 marks the completion of the first five years of the libraries’ Data and GIS Services Department. In 2009, when Mark Thomas and I formed the department, the name accurately reflected our staffing and services as Mark focused on GIS-related issues and I focused on data-related issues. As an increasing number of scholars have embraced data-driven research over the last five years , our services and staff have grown to support an increasingly diverse set of research needs at Duke.

In 2010-2011 academic year, the Libraries launched services around data management and sharing plans in anticipation of new funding rules surrounding research data. In 2012, the library expanded data services in collaboration with OIT’s Research Computing to offer one of the first data visualization consulting positions in the country. In 2013 and 2014, we expanded services and staff to include consultations on research computing and big data.

At this year’s Data and GIS Services annual retreat, we decided that the time has come to change the name of the department to reflect the broader range of staff and consulting services available. While we continue to support our traditional dimensions of data and GIS research, we intend to support a range of data needs across the following five themes:

Data and Visualization Services Themes

Data Sources
Get the data you need. Data and Visualization Services consultants can help you locate and license a diverse range of data sources. We also provide long term storage for Duke data collections through Duke’s institutional repository.

Data Storage and Management
Need help on a data management plan, want advice on archiving, or struggling with “big data” analytics? We are happy to consult!

Data Cleaning and Analysis
From Google Refine to the command line, we can help with data cleaning and analysis.

Mapping and GIS
Mapping and spatial analysis remain a core service for the data and visualization program.

Data Visualization
Our data visualization service can help with the most effective way to represent your data for both analysis and communication.

We appreciate the research community’s support as we’ve grown over the last five years. We look forward to working with you on a larger range of data challenges in the future!

Data Curation, Data Visualization, rstats, Statistics

Top 10 List – Data and GIS Edition

2014-05-21 Joel Herndon, Ph.D. Leave a comment

As we begin our summer in Data and GIS Services, we spend this post reflecting back on some of the services, software, and tools that made data work this spring more productive and more visible. We proudly present our top 10 list for the Spring 2014 semster:

10. DMPTool
While we enjoy working directly with researchers crafting data management plans, we realize that some data management needs arise outside of consultation hours. Fortunately, the Data Management Planning Tool (DMPTool) is there 24/7 to provide targeted guidance on data management plans for a range of granting agencies.

9. Fusion Tables
A database in the cloud that allows you to query and visualize your data, Fusion Tables has proven a powerful tool for researchers who need database functionality but don’t have time for a full featured database. We’ve worked with many groups to map their data in the cloud; see the Digital Projects blog for an example. Fusion Tables is a regular workshop in Data and GIS.

8. Open Refine
You could learn the UNIX command line and a scripting language to clean your data, but Open Refine opens data cleaning to a wider audience that is more concerned with simplicity than syntax. Open Refine is also a regular workshop in Data and GIS.

7. R and RStudio
A programming language that excels at statistics and data visualization, R offers a powerful, open source solution to running statistics and visualizing complex data. RStudio provides a clean, full-featured development environment for R that greatly enhances the analysis process.

6. Tableau Public
Need a quick, interactive data visualization that you can share with a wide audience? Tableau Public excels at producing dynamic data visualizations from a range of different datasets and provides intuitive controls for letting your audience explore the data.

5. ArcOnline
ArcGIS has long been a core piece of software for researchers working with digital maps. ArcOnline extends the rich mapping features of ArcGIS into the cloud, allowing a wider audience to share and build mapping projects.

4. Pandas
A Python library that brings data analysis and modeling to the Python scripting language, Pandas brings the ease and power of Python to a range of data management and analysis challenges.

3. RAW
Paste in your spreadsheet data, choose a layout, drag and drop your variables… and your visualization is ready. Raw makes it easy to go from data to visualization using an intuitive, minimal interface.

2. Stata 13
Another core piece of software in the Data and GIS Lab (and at Duke), Stata 13 brought new features and flexibility (automatic memory management — “hello big data”) that were greatly appreciated by Duke researchers.

1. R Markdown
While many librarians tell people to “document your work,” R Markdown makes it easy to document your research data, explain results, and embed your data visualizations using a minimal markup language that works in any text editor and ties nicely into the R programming language. For pulling it all together, R Markdown is number one in our top ten list!

We hope you’ve enjoyed the list! If you are interested in these or other data tools and techniques, please contact us at askdata@duke.edu!

Data Curation, MOOC, rstats, Statistics

Scaling Support: Designing Data for a Growing Statistics Program

2014-02-20 Joel Herndon, Ph.D. Leave a comment

r_stats101 How do you support 57,860 online students learning R and statistics ? Late last fall, Data and GIS Services shared this challenge with Professor Mine Çetinkaya-Rundel and the staff of CIT as we sought to translate Professor Çetinkaya-Rundel’s successful Statistics 101 course to a Coursera class on Data Analysis and Statistical Inference. While Data and GIS Services has supported Statistics 101 students for several years identifying appropriate data and using the R statistical language for their assignments, the scale of the Coursera course introduced new challenges of trying to provide engaging data to a very large audience without having the opportunity to provide direct support to everyone in the class.

In our initial meetings with Professor Çetinkaya-Rundel, she requested that Data and GIS create data collections for the course that would provide easy access in R and would include a range of statistical measures that would appeal to the diverse audience in the class. The first challenge — easy access to R — required some translation work. While R excels in its flexibility, graphics, and statistical power, it lacks some of the built in data documentation features present in other statistical packages. This project prompted Data and GIS to reconsider how to provide documentation and pre-formatted R data to an audience that would likely be unfamiliar with R and data documentation.

The second challenge — finding data that covered a wide range of interesting topics — proved much easier. The General Social Survey with its diverse and engaging questions on a wide range of topics proved to be an easy choice for the class. The American National Election Studies, also offered a diverse set of measures of public opinion that suited the course well. With these challenges identified and addressed, we spent the end of 2013 selecting portions of the data for class (subsetting), abridging the data documentation for instructional use, and transforming the data to address its usage in an online setting (processing missing values for R, creating factor variables).

As Professor Çetinkaya-Rundel’s class launches on February 17th, this project has given us a new appreciation of providing data and statistical services in a MOOC while also building course materials that we are using in Statistics 101 at Duke. While students begin the Coursera course on Data Analysis and Statistical Inference, students in Professor Kari Lock Morgan’s Statistics 101 class will use these data in their on-campus Duke course as well. We hope that both collections will reduce some of the technological hurdles that often confront courses using R as well as improving statistical literacy at Duke and beyond.

Data Curation, Data Management, Data Visualization, Duke research, GIS, spatial humanities, Statistics, workshops

Data and GIS Services Spring 2014 Workshop Series

2013-12-09 Joel Herndon, Ph.D. Leave a comment

Explore network analysis, text mining, online mapping, data visualization, and statistics in our spring 2014 workshop series. Our workshops provide a chance to explore new tools or refresh your memory on effective strategies for managing digital research. Interested in keeping up to date with workshops and events in Data and GIS? Subscribe to the dgs-announce listserv or follow us on Twitter (@duke_data).

Currently Scheduled Workshops

 Thu, Jan 9 2:00 PM – 3:30 PM  Data Management Plans – Grants, Strategies, and Considerations

 Mon, Jan 13 2:00 PM – 3:30 PM Webinar: Social Science Data Management and Curation

 Mon, Jan 13 3:00 PM – 4:00 PM Google Fusion Tables

 Tue, Jan 14 3:00 PM – 4:00 PM Open (aka Google) Refine 

 Wed, Jan 15 1:00 PM – 3:00 PM Stata for Research

 Thu, Jan 16 3:00 PM – 5:00 PM Analysis with R

 Tue, Jan 21 1:00 PM – 3:00 PM Introduction to ArcGIS

 Wed, Jan 22 1:00 PM – 3:00 PM ArcGIS Online

 Wed, Jan 22 3:00 PM – 4:00 PM Open (aka Google) Refine 

 Mon, Jan 27 2:00 PM – 3:30 PM Introduction to Text Analysis

 Wed, Jan 29 1:00 PM – 3:00 PM Analysis with R

 Thu, Jan 30 2:00 PM – 4:00 PM Stata for Research

 Mon, Feb 3 1:00 PM – 2:00 PM  Data Visualization on the Web

 Mon, Feb 3 2:00 PM – 3:00 PM  Data Visualization on the Web (Advanced)

 Tue, Feb 11 2:00 PM – 4:00 PM Using Gephi for Network Analysis and Visualization

 Wed, Feb 12 1:00 PM – 3:00 PM Introduction to ArcGIS

 Tue, Feb 18 2:00 PM – 3:30 PM Introduction to Tableau Public 8

 Tue, Feb 25 1:00 PM – 3:00 PM ArcGIS Online

 Thu, Feb 27 1:00 PM – 3:00 PM Historical GIS

 Mon, Mar 3 2:00 PM – 3:30 PM  Designing Academic Figures and Posters

 Tue, Mar 4 1:00 PM – 3:00 PM  Useful R Packages: Extensions for Data Analysis, Management, and Visualization

Data Curation, Data Management, Data Visualization, GIS, rstats, Statistics, Uncategorized

Data and GIS Fall 2013 Newsletter

2013-08-26 Joel Herndon, Ph.D. 1 Comment

Analyze, discover, manage, map, and visualize your data with Duke Libraries Data and GIS Services. Our team of five consultants provides a broad range of support in areas ranging from data analysis, data visualization, geographic information systems, financial data, statistical software and data storage and management. Our lab provides 12 workstations with the latest data software and three Bloomberg Professional workstations nearly 24/7 for the Duke community.

Data and GIS Workshop Series

All are welcome to the Data and GIS Workshop Series. Analyze, communicate, clean, map, represent and visualize your data with a wide range of workshops on data based research methods and tools. Details and registration for each class are available at the links that follow. (Interested in keeping up to date with workshops and events in Data and GIS? Just go to https://lists.duke.edu/sympa/info/dgs-announce and click on the “Subscribe” link at the bottom left.)

    Tue, Sep 3, 2013      1:00 PM - 3:00 PM    Introduction to ArcGIS    
    Wed, Sep 4, 2013     10:00 AM - 11:30 AM   Stata for Research    
    Wed, Sep 11, 2013    10:00 AM - 11:00 AM   Open (aka Google) Refine     
    Thu, Sep 12, 2013     1:00 PM - 3:00 PM    Analysis with R    
    Tue, Sep 17, 2013     1:00 PM - 2:30 PM    Introduction to Tableau Public 8    
    Thu, Sep 19, 2013    10:00 AM - 11:00 AM   Google Fusion Tables    
    Mon, Sep 23, 2013     1:00 PM - 2:30 PM    Introduction to Tableau Public 8    
    Tue, Sep 24, 2013     1:00 PM - 2:30 PM    Stata for Research    
    Mon, Sep 30, 2013    10:00 AM - 11:00 AM   Top 10 Dos and Don'ts for Charts and Graphs    
    Mon, Sep 30, 2013     1:00 PM - 3:00 PM    Introduction to ArcGIS    
    Tue, Oct 8, 2013      1:00 PM - 2:30 PM    Introduction to Text Analysis    
    Thu, Oct 10, 2013     1:00 PM - 3:00 PM    ArcGIS Special Topics: Geocoding & Proximity Analysis    
    Thu, Oct 17, 2013     1:00 PM - 3:00 PM    Historical GIS    
    Mon, Oct 28, 2013     1:00 PM - 2:00 PM    Designing Academic Figures and Posters    
    Tue, Oct 29, 2013     1:00 PM - 3:00 PM    Web GIS Applications

Data and GIS also offers instruction tailored to courses or research teams. Please contact askdata@duke.edu to schedule a session!

Data Management

Data Management Planning – DMPTool – Get 24/7 online help for your next data management plan, including information about Duke resources available for your data work.

Statistical Software Updates

Stata 13 is now available in our lab and training room
QSR’s Nvivo qualitative software is available on all machines
3 Bloomberg Professional Workstationsare available for individual or group use

Explore all of our Data and GIS Lab resources on our site at http://library.duke.edu/data/about/lab.html or come visit us on the second floor of Perkins Library.

Job Opportunities in Data and GIS Services

Data & GIS Services is hiring! We have two open positions for student web programmers interested in working on data visualization projects. See the Library Student employment page (http://library.duke.edu/jobs/students.html) for more information on how to apply. (The job can be found by searching for requisition number “DUL14-AMZ02”.)

New Data and Map Collections

CPS on Web (CPS Utilities Online)
CPS on Web is a set of utilities enabling you to access CPS data and documentation from this website. You may make tables and graphs from the CPS data, download data extractions, make estimations, get summaries and statistical measures, search the documentation, and make your own variables as functions of the existing ones.

Global Financial Data
Global Financial Data is a collection of financial and economic data provided in ASCII or Excel format. Data includes: long-term historical indices on stock markets; Total Return data on stocks, bonds, and bills; interest rates; exchange rates; inflation rates; bond indices; commodity indices and prices; consumer price indices; gross domestic product; individual stocks; sector indices; treasury bill yields; wholesale price indices; and unemployment rates covering over 200 countries.

LandScan Global
The LandScan Global Population Database provides global population distribution in a gridded GIS format at 30 arc-second resolution (approximately 1×1 km cells). Oak Ridge National Laboratory developed modeling techniques to disaggregate and interpolate census data within administrative boundaries to create a GIS layer showing population distribution as accurately and as timely as possible. EastView provides this data to use in GIS software as a WMS (Web Mapping Service) or as a WCS (Web Coverage Service) to allow a user to incorporate population distribution into GIS mapping and analysis.

Contact Us

email: askdata@duke.edu
twitter: duke_data or duke_vis