Introduction Weather data often comes in an untidy format that is suitable for looking at, but not so suitable for doing any kind of analysis with…

Eric is a Product Manager at Google working on Cloud Dataprep and recently Cloud Dataflow. Previously he was at Amazon Web Services, Harvard Business School, General Electric and University of Utah. He’s from Salt Lake City, Utah and lives in Mountain View, California with and wife and three kids…

IT IS THANKSGIVING WEEK HERE IN THE UNITED STATES. I’m getting ready to go out for a nice casual drive down Interstate I-95. Should be fun. After I get back stuffed with turkey and whatnot, we’ll get back to data visualizations and analysis…

Here are some notes on how the six features can be achieved using nothing else but “stuff on the internet”. I won’t provide detailed arguments, but just quick notes on how each of these features can already be implemented with minimal cost to researchers…

Match of the DayFollowing a defeat to North London rivals, Spurs now talking top 4, not title Pulis is outStodgy football and a points-per-game average lower than Steve Clarke will do that for you. His spell at WBA ended with the lowest ppg average of any of his three clubs…

The most widely used service for Geocoding is Google Maps, but you’d hit the limits early in your project if you don’t have a paid account. I often place data on a map, so I need precise geocoding. It’s not unlikely you had to test some other geocodig options ‘til you get the budget for a Google Maps paid account…

I read a lot of discussions about complexity in networking and IT today that include a large amount of FUD…

Last week I was working on some projects involving Posix and R code. I tend to use Atom Editor for such code instead of Rstudio, and sometimes I think the one an only valid contender against Rstudio is Atom Editor…

Introduction We are very accustomed to journey planners that help us navigate across the city, the country or the world. Google’s transit directions are an example: you enter a start and an end point, and it tells you which vehicles to get and where to transfer to another one…

My first post is going to be trivial but, I think, quite mesmerizing. This is sweet, but something is missing… perhaps there are not enough points. Let’s use 10,000,000! That’s more like it. Now let’s make another attractor, using 10,000,000 magenta points…

This month I reached the dreaded “2 hours limit” at Docker Hub on building my Docker images. The images are built but the message shown is “build cancelled”. You can still use it, but on the front desk, seems that I’m not doing a good job. Clearly, something to fix…

For miles down I-40 you’ll see billboards for this restaurant and brewery advertising a free 72oz steak, if you can eat it. An office debate erupted amongst two of my colleagues over just what types of people end up beating the beast. They argued. I fired up rvest…

In this article we’ll take a look at another Angular abstraction - the differs and more specifically the IterableDiffer; we’ll explain what the differs are and how the framework uses them internally…

People have different reasons to give talks. For me the main motives are: All that showed how a supportive community can make a difference and help more people get engaged…

EDIT: I provided a quick update here, and it will be the main entrance of a post series regarding this subject, cheers ! https://roelandtn.frama.io/post/update-for-the-mexico-project/ Hello there ! It has been a long time since my last post…

TLDR: If you can assume close fit for the RMSEA, there is no reason why you cannot for a Chi-Square test in SEMs…

COME AND MAKE A MAJESTIC MORTGAGE RATE PLOT WITH ME. We’ll use R to plot a few visualizations of mortgage rates. I recently gave a number of talks about the economic outlook and housing. One point I like to make is that mortgage rates are low. I’ve shown this through a variety of visualizations…

Certain applications lend themselves to pure parallel computation better than others. In some cases we require to apply certain algorithms over a “window” in our data…

When we are trying to assess the effect of an exposure or intervention on an outcome, confounding is an ever-present threat to our ability to draw the proper conclusions…

Install the latest version with: One common idiom for dealing with this problem is to save the current state, make your change, then restore the previous state. However this approach can fail if there’s an error before you are able to reset the options. Here are some highlights of new functions for v2.1.0…

If you’ve been an information security practitioner for more than a few years, you’ve likely witnessed your share of disappointing purchases, implementations and initiatives…

Colt McAnlis is a Developer Advocate at Google focusing on performance & compression. Before that, he was a graphics programmer in the games industry working at Blizzard, Microsoft (Ensemble), and Petroglyph…

’Tis the season for white elephant / גמד וענק / Yankee swap / secret santa-ing! There are various rules for this, for our version: 🏷 each participant receives the name of someone else to purchase a gift for 🎁 gifts are exchanged at a party 🤔 the receiver is tasked with guessing who the gift-giver was! We thought it’d be particularly fun to do it #rstats…

The latest version of tidyverse can be installed…

The problem with this approach, is that you cannot reuse any of the code there, even if you put it inside a…

This week we have a guest post written by Hanee’ Medhat Hanee’ is a Big Data Engineer, with experience working with massive data in many industries, such as Telecommunications and Banking…

In this post, I’d like to share some work related to geo-spatial mapping, statewide school ratings, and US Census Bureau data using census tracts…

In this post, we’ll focus on techniques from functional programming we can apply to improve the performance of our applications, more specifically pure pipes, memoization, and referential…

I’ve given a version of this talk at all three EARL conferences this year starting in San Francisco…

Thanksgiving 🦃 is right around the corner 🎉 – this year we are hosting 17 people 😱…

White boxes and their impact on enterprise networking is a hot topic today, with many point of…

On AngularConnect 2017 in London, I gave a talk called “Purely Fast.” In the presentation, I showed how step by step we can improve the performance of a business application…

#Pre-requisites Before we begin, it’s a good idea to install some general purpose tools that will help us out once R is installed. The first of these dependencies are a bunch of development tools…

When something goes wrong it’s often tempting to attribute it to a single root cause: The site was down because the database crashed. The application broke because the developer pushed bad code. The machine stopped responding because it ran out of disk space…

How to extract data from PDFs (in this case, homelessness figures in London), alongside a showcase of algorithmic tesselation of geospatial…

Please do share your data analyst meme creations.…

Photo by Rohit Tandon on Unsplash A few years ago I decided to transition career and become a professional web developer. Married, with a young son, and a full time job as a management consultant (i.e. long hours and lots of travel) this was not an easy undertaking…

Currently I am studying for my qualifying exams on which the topic is “using deep neural networks for classification of time series data…

Brian Granatir has been developing for the cloud since the beginning, back in 2007. He left Oregon and moved to New Zealand to be with his future wife in 2014. In 2017, he joined Smart Parking to help with the development of their new Smart City platform built on GCP…

HEY! HERE IS A VIDEO SHOWING HOUSE PRICE TRENDS around the United States. Earlier this year we looked at how to get the data and plot it using R. I made the video using the PowerPoint to .mp4 workflow I outlined here. Below I’ll review how to build this file…

Or get the development version: If the instance uses SSL, simply specify that like: This change does break the interface we had in the old version, but we think it’s worth it…

The ordinal data is generated after a data set has been created with an adjustment variable. We have to provide the data.table name, the name of the adjustment variable, and the base probabilities. That’s really it…

You may wonder why this can’t be written like this: Let me explain a bit…

First, let’s load the data: .. and do some data cleaning: change column names, get rid of the order in factor levels and remove rows with empty cells: That’s better! Now, let’s set up the local H2O instance… ..…

I want to try and probe a question that was raised since Las Vegas and now revived due to the tragedy in Sutherland Springs,TX: Given the free trade between states, can a state unilaterally regulate firearms…

LET’S TAKE A LOOK AT RECENT LABOR MARKET TRENDS IN THE UNITED STATES. Below we’ll plot labor market trends using the U.S. Bureau of Labor Statistics Job Openings and Labor Turnover Survey (JOLTS). Last year we looked at how to get the data and plot it using R…

… data sets regarding songs on the Billboard Hot 100 list from 1960 to 2016, including ranks for the given year, musical features, and lyrics. So, this blogpost walks through how you might start to unpack the data, clean it, and draw some interesting conclusions. I also wanted to avoid the “draw the rest of the fucking owl”…

WE ARE LATE FOR HALLOWEEN, but let’s get out our broom and purrr as we tidy some statistical results. Today I had occasion to be reminded of competing risks and a handy statistical result on competing risks from A.P. Basu and J.K…

Match of the DayMourinho still can’t buy a result away against big clubs The Spanish connectionMorata connected up with Azpilicueta’s cross for Chelsea’s winner against Manchester United and he has now assisted on five of the striker’s seven league goals this season, only topped by six of the 52 goals Diego Costa scored for the…

Applying linear programming principles to the selection of a fantasy football…

Today I passed exam 2V0-642 to update my VCP5-DCV and got VCP6-NV (NSX v6.2) I’ll share here what I used to prepare the exam. Learning Path I passed exam VCP5-DCV in 2015. At the time I expected to work more with datacenter technology and I needed a more in-depth knowledge of how virtualization works…

LET US REVIEW HOUSING MARKET TRENDS in the United States through the first three quarters of 2017. Economic background The overall economic environment remains favorable for housing. Interest rates are low, the labor market has been solid and income growth, while modest, has begun to tick up…

This is the first of a series of posts to introduce my new esvis R package, why I think it’s important, and some of its capabilities. As of this writing the current version on CRAN is 0.0.1, so it’s obviously still fresh and may have some bugs…

I’m currently working on a book for Packt publishing titled Serverless Design Patterns and Best Practices. While writing and whipping out tons of examples is quite a bit of work, and I sometimes curse myself for agreeing to this, I’m quite excited as I work through the chapters and as it comes together…

In this post I’m going to run through a brief example of using sparklyr in R. This package provides a way to connect to Spark from within R, while using the dplyr functions we all know and love. I was entirely new to Spark, and databases in general, before having a play with sparklyr…

I’m Vid Jain, CEO & Founder of Wallaroo Labs. I’m writing today to tell you about how we open-sourced Wallaroo, our software framework for data processing, and the lessons we learned along the way. Our engineering team was experienced at writing great software, but now we faced a new set of challenges…

It’s time to learn everything about Cloud IoT Core from Indranil Chakraborty, Product Manager, and Gabe Weiss, Developer Advocate on IoT. Indranil is a product manager for Google Cloud Platform and leads product strategy and development for Cloud IoT Core. Previously, he held product management roles at Google Fiber and sales strategy roles for Google AdWords…

Here’s the scenario: we have an intervention that we think will improve outcomes for a particular population…

Lubridate is a package that makes working with date-time and time-span objects easier. It provides fast and user friendly parsing of date-time strings, extraction and updating of components of a date-time objects (years, months, days etc.) and algebraic manipulation on date-time and time-span objects…

What we created was a bunch of awesome individuals from 12 different countries. Before the ozunconf we discussed and dreamt up projects to work on for a few days, then met up and brought a few of them into reality. Before the ozunconf, we discussed various ideas for projects in GitHub…

Match of the DayKane-less Spurs lose to Pogba-less United and trail City by 8 points Final Top 6?The accepted ‘top six’ i.e Chelsea, Spurs, Man City, Liverpool, Arsenal and Man Utd…

However, over the last year or so, a couple of things started to annoy me so much that I stopped enjoying writing posts on Blogger. For example, there is no easy way to bulk edit older posts (I wanted to change links from files hosted on public Dropbox folders elsewhere), or to write posts in Markdown or better RMarkdown…

Install the latest version with: glue is also vectorized over its inputs…

However, we almost always want to discuss our inference about the population, not individual subjects. Further, if we wish to discuss individual subjects, we should place them in the context of other subjects…

I PUT TOGETHER SOME SLIDES SUMMARIZING our recent work on dynamic model averaging. See here and here for more blah blah blah. See below for some slides. Click here for a fullscreen version here. Making the Preso Let me also share with you the R code I used to generate these slides…

This post serves a few purposes: Exercising Deep Learning with Regression, as opposed to Classification, which is my comfort zone. A tribute to my previous working place - ebay. Trying to combine text AND image data in a “Bi-Modal” model - never did it before. Doing something which involves Fashion…

Fermat and his library This morning I woke up to a delightful tweet from fermatslibrary about sample random uniform numbers and how many it takes, on average, to sum to 1…

BACK WE GO INTO THE VASTY DEEP. LAST TIME we introduced the idea of using dynamic model averaging to forecast recessions. I was so excited about the new approach that I didn’t take the time to break down what was going on with it…

When a customer calls with a problem or request I often see a chance to investigate a technology, learn something new or apply random skills to find a creative solution. This time is about an ASA, customer noticed too much traffic on the Internet facing interface…

TLDR: If you have good measurement quality, conventional benchmarks for fit indices may lead to bad decisions. Additionally, global fit indices are not informative for investigating misspecification. I am working with one of my professors, Dr…

Last Tuesday (October 24th) I was at Madrid R User Group to give a tech talk about using Docker for automating our setup with Rstudio. Using Rocker images for easy and quick deployment. Here, the slides hoping someone will find it useful.…

One of my journalist friend was looking at the result of the local election in Luxembourg and he was dissatisfied because he was unable to compare the results of all the communes…

I’ve had to change my R package names a few times. Every time I’ve had to do this, there were a few things I had to remember to do. Here’s a blog post that describes how to do that. When you change your package name, here is a list of things you need to do. And that’s basically it! Hooray, now to handle the git…

I have rather belatedly gotten around to viewing the adult animated series, Archer, on Netflix…

First, let’s load some packages: And it is also possible to plot the effects with base graphics: Which makes it possible to extract the effects for a list of individuals that you can create…

HERE THE LITERATURE IS VASTY DEEP. In this post we’ll dip our toes, every so slightly, into the dark waters of macroeconometric forecasting. I’ve been studying some techniques and want to try them out. I’m still at the learning and exploring stage, but let’s do it together. In this post we’ll conduct an exercise in forecasting U.S…

mediumr allows you to knit and post R Markdown to Medium. You can install mediumr from github with: The addin knits the Rmd and shows the preview dialog…

Hi there! Today, I want to talk to you about why we chose to write Wallaroo, our distributed data processing framework for building high-performance streaming data applications, in Pony. It’s a question that has come up with some regular frequency from our more technically minded audiences…

This lets us interact with R and Slack in new ways, by getting active updates on long simulations directly to your (and your team’s) mobile device and multitask away from your computer…

LET’S LOOK AT NEW HOME SALES. Today the U.S. Census Bureau joint with the Department of Housing and Urban Development (HUD) released new home sales estimates through September of 2017…

Compiling all Rmd files is “safe” in the sense that we can notice if some Rmd becomes impossible to compile due to some breaking changes of some package. But, it may be time-consuming and can be a problem for those who have a lot of .Rmd files…

Compare this to an adjacency matrix with the same data…

Google, the Cloud, or podcasts would not exist without the internet, so it’s with an incredible honor that we celebrate our 100th episode with one of its creators: Vint Cerf…

In this day and age of so many sharing services like Uber and Lyft , pricey hotels are being replaced by Airbnb…

Some of my friends didn’t aware that dplyr now accepts characters…

The key updates…

jsTree is a R package that is a standalone htmlwidget for the jsTree JavaScript library. It can be run from the R console directly into the Rstudio Viewer, be used in a Shiny application or be part of an Rmarkdown html output…

These conditional distributions for historical and projected temperature and precipitation over different geographic regions, time periods, climate models and greenhouse gas emissions scenarios represent the source data sets available in the app…

LET’S MAKE SHARING BETTER ON THIS PAGE. I saw this today: 💫 how-to for making your #blogdown social-friendly: “Socialize your blogdown” by @xvrdm https://t.co/RurfUvRf6X #rstats #opengraph pic.twitter…

We are generally most interested in finding out where the peak of that curve is, because the parameters associated with that point (the maximum likelihood estimates) are often used to describe the “true” underlying data generating…

Seasonality and Trends Time-series analysis is a battle on multiple fronts by definition. One has to deal with (dynamic) trends, seasonality effects, and good old noise…

IN THIS POST I WANT SHARE A METHOD FOR MAKING SMOOTH POWERPOINT ANIMATIONS USING R…

I recently discover a R package, called liftr package. It allows the build of pdf document (and html files, but i didn’t test it) from a Rmarkdown file. Fully integrated in RStudio, the R code is executed (and other code as well, I tried python) and results are displayed…

You may notice that diffs of .Rd files are suppressed by default on GitHub since some time. Do you wonder who did this? It’s me, yay! This is my pull request: Though I thought I’ve done the right thing at that time, now I’m afraid this change may be bad for some cases… After the relese of roxygen2 6.0.0, the game has changed a bit…

WE ARE ON OUR WAY TOWARDS BUILDING a tidy PowerPoint workflow. In this post I want to build on my earlier posts (see here for an introduction and here for a more sophisticated approach) for building a PowerPoint presentation with R and try to make it even purrrtier…

I wrote a really quick script to install R packages from the command line that I thought I’d share. It doesn’t really do a great deal, but you can use it to install one package at a time. Save the below as rpkginstall and make sure it’s executable with chmod + x rpkginstall. Then you can install a package like this example, which would install dplyr, …

In episode 13 of the Network Collective podcast around minute 26 Jordan Martin asks: Aren’t we all just following a trend? The discussion topic is how to mentor juniors in a learning path to grow their skills and be experts…

I am completely obsessed with Shiny and these days I end up presenting most of my work in a Shiny app. If it’s not worth putting in a Shiny app it’s not worth doing. Getting started with Shiny is actually a lot easier than a lot of people make it out to be…

Scaling stateful applications is hard. As your business grows, you’re eventually going to find that demand is greater than capacity. That means you can’t simply deploy your application to a set number of servers and forget it…

MORTGAGE RATES ARE LOW IN THE UNITED STATES. How low? Let’s take a look. We’ll use R to plot a few visualizations of mortgage rates. We’ll also try out some of the nice features in the tibbletime package that help when working with time series data…