Why?

July 1, 2015

useR 2015: Computational

Filed under: R, useR 2015 — csgillespie @ 12:19 pm

These are my initial notes from useR 2015. I will/may revise when I have time.

Computational Performance; Chair: Dirk Eddelbuettel

Running R+Hadoop using Docker Containers (E. James Harner)

Introduction

  • Big data architectures:
    • HDFS/Hadoop: software framework for distributed storage and distributed processing
    • Tachyon/Spark: uses in-memory

Rc2 server (R cloud computing)

  • Has an editor & output panel. Interactive collaboration (Demo)
  • highly scalable
  • 4-tier architecture: client, app server, compute cloud (JSON over BSD sockets for R),
    databases (pgSQL & couchdb)

RC2 Client

  • Sharable project and workspaces
  • Graphs are written to files and moved to the database as blobs
  • Security: A 3 value token is used for auto-logins

Summary

Rc2 is an accessible IDE for students and data scientist to allow real time collaboration. It also acts as a front end to Hadoop and Spark clusters.

Algorithmic Differentiation for Extremum Estimation: An Introduction Using RcppEigen (Matt P. Dziubinski)

Why

  • Parametric model: We want to estimate a parameter by maximizing an objective function
  • No closed formed expressions, so we need to numerically optimize

Algorithms

  • Derivative free: does not rely on knowledge of the objective function
  • Gradient-based: needs the gradient of the objective function
    • Steepest ascent, newton
    • Often exhibit superior convergence rates
    • But getting the gradient can be tricky, e.g. finite difference methods

Algorithmic diffentiation

  • Essentially use the chain rule
  • Need to recode the objective function in Cpp using Rcpp

Improving computational performance with algorithm engineering (Kirill Müller)

Application: activity based microsimulation models

Weighted sampling without replacement

  • Random sample: sample.int
  • Common framework: Subdivide an interval according to probabilities
    • If sampling without replacement, remove sub-interval
  • R uses trivial algorithm with update in O(n)
    • Heap-like data structure
  • Alternative approaches:
    • Rejection sampling
    • One-pass sampling (Efraimidis and Spirakis, 2006)

Statistical matching (data fusion)

  • Use Gower's distance to compare distribution
    • works with interval, ordinal and nominal variables

Please note that the notes/talks section of this post is merely my notes on the
presentation. I may have made mistakes: these notes are not guaranteed to be
correct. Unless explicitly stated, they represent neither my opinions nor the
opinions of my employers. Any errors you can assume to be mine and not the
speaker’s. I’m happy to correct any errors you may spot – just let me know!

Advertisements

useR 2015: Networks

Filed under: R, useR 2015 — csgillespie @ 9:39 am

These are my initial notes from useR 2015. Will revise when I have time.

fbRads: Analyzing and managing Facebook ads from R (Gergely Daroczi)

Modern advertising

Google/Amazon/Facebook use our information

Ad platforms: Google: RAdwords, facebook likes: fbRads. You can use the facebook API to get information from facebook. Get hashes of email address, not the actual address. In the last few years, the API has changed.

Example

  1. Grab useR's email addresses from CRAN and R-help mailing list.
  2. Create a facebook app with API to get a token.
  3. Create a custom audience.
  4. Create lookalike audiences: get facebook users who are similar to my target list.
  5. Define audience, ad and budget.
  6. Upload an image and description.
  7. Run A/B testing.

The performance metrics API is still being developed.

Web scraping with R – A fast track overview. (Peter Meißner)

There are a number of R packages for web-scraping.

Two problems:

  • Download: protocols/procedures, i.e. HTTP, cookies, POST, GET
  • extraction: parsing/extraction/cleansing, i.e. XML, JSON, html into R

Reading text from the Web

The simplest solution is to use readLines, then use some regular expressions (either with base R or stringr or …).

Reading HTML/XML

Use rvest and use xml_structure to view the structure of the XML scheme. To extract text, we need to use XPath (still using rvest). Within rvest there are a number of convenience functions, e.g. html_table to get a list of tables.

JSON

Use jsonlite to translate JSON to a data frame.

HTML forms/HTTP methods

Use httr and rvest packages.

Overcoming the Javascript Barrier

Use RSelenium for browser automation

Conclusion

  • Don't use Windows for web scraping. Use Linux (or if you must, a Mac)
  • Start with stringr, rvest, jsonlite
  • Need to learn regular expression, file manipulation
  • Before scraping, look for the download button

multiplex: Analysis of Multiple Social Networks with Algebra (Antonio Rivero Ostoic)

Motivation

  • multiplex is a package designed to perform algebraic analyses of multiple networks (but isn't limited to algebra)
  • The function zbind creates multivariate network data from arrays
  • perm manipulates network data

Two-mode networks are represented in a Galois framework. This makes analysis easier(?)

What's new in igraph and networks (Gabor Csardi)

Abstract

igraph is the premier R package for the analysis of network data and it went through major restructuring recently and has changed a lot since last time it was featured at useR! in 2008. This talk introduces the new/updated features of igraph: – Simplified ways of graph manipulation. – New methods community detection. – New layouts for graph visualization. – New statistical methods: graphlets, embeddings, graph matching, cohesive blocks, etc. – How to use igraph graphs with new visualization tools: DiagrammeR, D3, etc.

The igraph package deals mainly with infrastructure. It's actually a C library, with an R and python interface.

What's new: [ and [[

The [ operator makes the graph behave like an adjacency matrix. For example, to check if an edge exists, use air["BOS", "SFO"]. Can also use it to manipulate the network, e.g. to add or remove edges.

The [[ can be used to get all adjacent vertices

What's new: consistent function names and manipulators

  • make_*, sample_*, cluster_, layout_*, graph_from_*
  • manipulators: make_ and sample_
  • Pipe friendly syntax
  • Easier connection to other packages, e.g. networkD3

Current work

  • Better connection to other packages
  • Inference
  • Infrastructure cleanup

Please note that the notes/talks section of this post is merely my notes on the
presentation. I may have made mistakes: these notes are not guaranteed to be
correct. Unless explicitly stated, they represent neither my opinions nor the
opinions of my employers. Any errors you can assume to be mine and not the
speaker’s. I’m happy to correct any errors you may spot – just let me know!

useR 2015: Romain Francois: My R adventures

Filed under: R, useR 2015 — Tags: , — csgillespie @ 8:05 am

Using R since 2002 and has been working on Rcpp, Rcpp11, Rcpp14 and dplyr
internals. Worked on a number of big projects.

  • 2005 he set up the R Graph Gallery
  • 2009 worked on rJava
  • 2010 Rcpp
  • 2013 dplyr

Key themes are Performance and usabililty

rJava 0.7-*

Creating objects was messy

d <-jnew("java/lang/Double", 42
.jcal(d, "D", "doubleValue)

rJava 0.8-*

d <- new(J("java/lang/Double"), 42)
d$doubleview

Also much easier to import java packages.

Rcpp

Suppose you have

double add(double a, double b){ return a+b;}

and you want to use it in R. This used to be a lot of work. Before Rcpp, you
used the R/C Api, i.e. use SEXP. A lot of work and boilerplate. With Rcpp the
number of characters needed to translate the simple function above went from 250
to 50. Around 66% of CRAN packages depend (on some way) on Rcpp.

RcppParallel (tbb: thread building blocks)

The package makes it much easier to run things in parallel. Amazingly, a simple
parallel version of sqrt is faster than sqrt

dplyr (everyone knows what it does)

Uses hybrid evaluation. Looking to bring RcppParallel in the (near?) future.

Please note that the notes/talks section of this post is merely my notes on the
presentation. I may have made mistakes: these notes are not guaranteed to be
correct. Unless explicitly stated, they represent neither my opinions nor the
opinions of my employers. Any errors you can assume to be mine and not the
speaker’s. I’m happy to correct any errors you may spot – just let me know!

Create a free website or blog at WordPress.com.