Introduction to rsolr

Michael Lawrence
November 21, 2026

1 Introduction

The rsolr package provides an idiomatic (R-like) and extensible interface between R and Solr, a search engine and database. Like an onion, the interface consists of several layers, along a gradient of abstraction, so that simple problems are solved simply, while more complex problems may require some peeling and perhaps tears. The interface is idiomatic, syntactically but also in terms of intent. While Solr provides a search-oriented interface, we recognize it as a document-oriented database. While not entirely schemaless, its schema is extremely flexible, which makes Solr an effective database for prototyping and adhoc analysis. R is designed for manipulating data, so rsolr maps common R data manipulation verbs to the Solr database and its (limited) support for analytics. In other words, rsolr is for analysis, not search, which has presented some fun challenges in design. Hopefully it is useful — we had not tried it until writing this document.

We have interfaced with all of the Solr features that are relevant to data analysis, with the aim of implementing many of the fundamental data munging operations. Those operations are listed in the table below, along with how we have mapped those operations to existing and well-known functions in the base R API, with some important extensions. When called on rsolr data structures, those functions should behave analogously to the existing implementations for data.frame. Note that more complex operations, such as joining and reshaping tables, are best left to more sophisticated frameworks, and we encourage others to implement our extended base R API on top of such systems. After all, Solr is a search engine. Give it a break.

Operation R function
Filtering subset
Transformation transform
Sorting sort
Aggregation aggregate

2 Demonstration: nycflights13

2.1 The Dataset

As part demonstration and part proof of concept, we will attempt to follow the introductory workflow from the dplyr vignette. The dataset describes all of the airline flights departing New York City in 2013. It is provided by the nycflights13 package, so please see its documentation for more details.

library(nycflights13)
dim(flights)
## [1] 336776     19
head(flights)
## # A tibble: 6 x 19
##    year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##   <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
## 1  2013     1     1      517            515         2      830            819
## 2  2013     1     1      533            529         4      850            830
## 3  2013     1     1      542            540         2      923            850
## 4  2013     1     1      544            545        -1     1004           1022
## 5  2013     1     1      554            600        -6      812            837
## 6  2013     1     1      554            558        -4      740            728
## # i 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>

2.2 Populating a Solr core

The first step is getting the data into a Solr core, which is what Solr calls a database. This involves writing a schema in XML, installing and configuring Solr, launching the server, and populating the core with the actual data. Our expectation is that most use cases of rsolr will involve accessing an existing, centrally deployed, usually read-only Solr instance, so those are typically not major concerns. However, to conveniently demonstrate the software, we need to violate all of those assumptions. Luckily, we have managed to embed an example Solr installation within rsolr. We also provide a mechanism for autogenerating a Solr schema from a data.frame. This could be useful in practice for producing a template schema that can be tweaked and deployed in shared Solr installations. Taken together, the process turns out to not be very intimidating.

We begin by generating the schema and starting the demo Solr instance. Note that this instance is really only meant for demonstrations. You should not abuse it like the people abused the poor built-in R HTTP daemon.

library(rsolr)

## Loading required package: BiocGenerics

## Loading required package: generics

##
## Attaching package: ’generics’

## The following objects are masked from ’package:base’:
##
## as.difftime, as.factor, as.ordered, intersect, is.element, setdiff,
## setequal, union

##
## Attaching package: ’BiocGenerics’

## The following objects are masked from ’package:stats’:
##
## IQR, mad, sd, var, xtabs

## The following objects are masked from ’package:base’:
##
## anyDuplicated, aperm, append, as.data.frame, basename, cbind,
## colnames, dirname, do.call, duplicated, eval, evalq, Filter, Find,
## get, grep, grepl, is.unsorted, lapply, Map, mapply, match, mget,
## order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
## rbind, Reduce, rownames, sapply, saveRDS, table, tapply, unique,
## unsplit, which.max, which.min

##
## Attaching package: ’rsolr’

## The following object is masked from ’package:stats’:
##
## ftable

## The following object is masked from ’package:base’:
##
## grouping

schema <- deriveSolrSchema(flights)
solr <- TestSolr(schema)

## Starting Solr...

## Use options(verbose=TRUE) to diagnose any problems.

## Solr not found

## Solr installation not found, installing...

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

## Solr started at: http://localhost:8983/solr/flights

Next, we need to populate the core with our data. This requires a way to interact with the core from R. rsolr provides direct access to cores, as well as two high-level interfaces that represent a dataset derived from a core (rather than the core itself). The two interfaces each correspond to a particular shape of data. SolrList behaves like a list, while SolrFrame behaves like a table (data frame). SolrList is useful for when the data are ragged, as is often the case for data stored in Solr. The Solr schema is so dynamic that we could trivially define a schema with a virtually infinite number of fields, and each document could have its own unique set of fields. However, since our data are tabular, we will use SolrFrame for this exercise.

sr <- SolrFrame(solr$uri)

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")

Finally, we load our data into the Solr dataset:

sr[] <- flights

This takes a while, since Solr has to generate all sorts of indices, etc.

As SolrFrame behaves much like a base R data frame, we can retrieve the dimensions and look at the head of the dataset:

dim(sr)

## [1] 336776     19
head(sr)
## DocDataFrame (6x19)
##   year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## 1 2013     1   1      517            515         2      830            819
## 2 2013     1   1      533            529         4      850            830
## 3 2013     1   1      542            540         2      923            850
## 4 2013     1   1      544            545        -1     1004           1022
## 5 2013     1   1      554            600        -6      812            837
## 6 2013     1   1      554            558        -4      740            728
##   arr_delay carrier flight tailnum origin dest air_time distance hour minute
## 1        11      UA   1545  N14228    EWR  IAH      227     1400    5     15
## 2        20      UA   1714  N24211    LGA  IAH      227     1416    5     29
## 3        33      AA   1141  N619AA    JFK  MIA      160     1089    5     40
## 4       -18      B6    725  N804JB    JFK  BQN      183     1576    5     45
## 5       -25      DL    461  N668DN    LGA  ATL      116      762    6      0
## 6        12      UA   1696  N39463    EWR  ORD      150      719    5     58
##             time_hour
## 1 2013-01-01 10:00:00
## 2 2013-01-01 10:00:00
## 3 2013-01-01 10:00:00
## 4 2013-01-01 10:00:00
## 5 2013-01-01 11:00:00
## 6 2013-01-01 10:00:00

Comparing the output above the that of the earlier call to head(flights) reveals that the data are virtually identical. As Solr is just a search engine (on steroids), a significant amount of engineering was required to achieve that result.

2.3 Restricting by row

The simplest operation is filtering the data, i.e., restricting it to a subset of interest. Even a search engine should be good at that. Below, we use subset to restrict to the flights to those departing on January 1 (2013).

subset(sr, month == 1 & day == 1)

## ’flights’ (ndoc:842, nfield:19)
##     year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##   1 2013     1   1      517            515         2      830            819
##   2 2013     1   1      533            529         4      850            830
##   3 2013     1   1      542            540         2      923            850
##   4 2013     1   1      544            545        -1     1004           1022
##   5 2013     1   1      554            600        -6      812            837
## ...  ...   ... ...      ...            ...       ...      ...            ...
## 838 2013     1   1     2356           2359        -3      425            437
## 839 2013     1   1     <NA>           1630      <NA>     <NA>           1815
## 840 2013     1   1     <NA>           1935      <NA>     <NA>           2240
## 841 2013     1   1     <NA>           1500      <NA>     <NA>           1825
## 842 2013     1   1     <NA>            600      <NA>     <NA>            901
##     arr_delay carrier flight tailnum origin dest air_time distance hour minute
##   1        11      UA   1545  N14228    EWR  IAH      227     1400    5     15
##   2        20      UA   1714  N24211    LGA  IAH      227     1416    5     29
##   3        33      AA   1141  N619AA    JFK  MIA      160     1089    5     40
##   4       -18      B6    725  N804JB    JFK  BQN      183     1576    5     45
##   5       -25      DL    461  N668DN    LGA  ATL      116      762    6      0
## ...       ...     ...    ...     ...    ...  ...      ...      ...  ...    ...
## 838       -12      B6    727  N588JB    JFK  BQN      186     1576   23     59
## 839      <NA>      EV   4308  N18120    EWR  RDU     <NA>      416   16     30
## 840      <NA>      AA    791  N3EHAA    LGA  DFW     <NA>     1389   19     35
## 841      <NA>      AA   1925  N3EVAA    LGA  MIA     <NA>     1096   15      0
## 842      <NA>      B6    125  N618JB    JFK  FLL     <NA>     1069    6      0
##               time_hour
##   1 2013-01-01 10:00:00
##   2 2013-01-01 10:00:00
##   3 2013-01-01 10:00:00
##   4 2013-01-01 10:00:00
##   5 2013-01-01 11:00:00
## ...                 ...
## 838 2013-01-02 04:00:00
## 839 2013-01-01 21:00:00
## 840 2013-01-02 00:00:00
## 841 2013-01-01 20:00:00
## 842 2013-01-01 11:00:00

Note how the records at the bottom contain missing values. Solr does not provide any facilities for missing value representation, but we mimic it by excluding those fields from those documents.

We can also extract ranges of data using the canonical window() function:

window(sr, start=1L, end=10L)

## DocDataFrame (10x19)
##    year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##  1 2013     1   1      517            515         2      830            819
##  2 2013     1   1      533            529         4      850            830
##  3 2013     1   1      542            540         2      923            850
##  4 2013     1   1      544            545        -1     1004           1022
##  5 2013     1   1      554            600        -6      812            837
##  6 2013     1   1      554            558        -4      740            728
##  7 2013     1   1      555            600        -5      913            854
##  8 2013     1   1      557            600        -3      709            723
##  9 2013     1   1      557            600        -3      838            846
## 10 2013     1   1      558            600        -2      753            745
##    arr_delay carrier flight tailnum origin dest air_time distance hour minute
##  1        11      UA   1545  N14228    EWR  IAH      227     1400    5     15
##  2        20      UA   1714  N24211    LGA  IAH      227     1416    5     29
##  3        33      AA   1141  N619AA    JFK  MIA      160     1089    5     40
##  4       -18      B6    725  N804JB    JFK  BQN      183     1576    5     45
##  5       -25      DL    461  N668DN    LGA  ATL      116      762    6      0
##  6        12      UA   1696  N39463    EWR  ORD      150      719    5     58
##  7        19      B6    507  N516JB    EWR  FLL      158     1065    6      0
##  8       -14      EV   5708  N829AS    LGA  IAD       53      229    6      0
##  9        -8      B6     79  N593JB    JFK  MCO      140      944    6      0
## 10         8      AA    301  N3ALAA    LGA  ORD      138      733    6      0
##              time_hour
##  1 2013-01-01 10:00:00
##  2 2013-01-01 10:00:00
##  3 2013-01-01 10:00:00
##  4 2013-01-01 10:00:00
##  5 2013-01-01 11:00:00
##  6 2013-01-01 10:00:00
##  7 2013-01-01 11:00:00
##  8 2013-01-01 11:00:00
##  9 2013-01-01 11:00:00
## 10 2013-01-01 11:00:00

Or, as we have already seen, the more convenient:

head(sr, 10L)

## DocDataFrame (10x19)
##    year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##  1 2013     1   1      517            515         2      830            819
##  2 2013     1   1      533            529         4      850            830
##  3 2013     1   1      542            540         2      923            850
##  4 2013     1   1      544            545        -1     1004           1022
##  5 2013     1   1      554            600        -6      812            837
##  6 2013     1   1      554            558        -4      740            728
##  7 2013     1   1      555            600        -5      913            854
##  8 2013     1   1      557            600        -3      709            723
##  9 2013     1   1      557            600        -3      838            846
## 10 2013     1   1      558            600        -2      753            745
##    arr_delay carrier flight tailnum origin dest air_time distance hour minute
##  1        11      UA   1545  N14228    EWR  IAH      227     1400    5     15
##  2        20      UA   1714  N24211    LGA  IAH      227     1416    5     29
##  3        33      AA   1141  N619AA    JFK  MIA      160     1089    5     40
##  4       -18      B6    725  N804JB    JFK  BQN      183     1576    5     45
##  5       -25      DL    461  N668DN    LGA  ATL      116      762    6      0
##  6        12      UA   1696  N39463    EWR  ORD      150      719    5     58
##  7        19      B6    507  N516JB    EWR  FLL      158     1065    6      0
##  8       -14      EV   5708  N829AS    LGA  IAD       53      229    6      0
##  9        -8      B6     79  N593JB    JFK  MCO      140      944    6      0
## 10         8      AA    301  N3ALAA    LGA  ORD      138      733    6      0
##              time_hour
##  1 2013-01-01 10:00:00
##  2 2013-01-01 10:00:00
##  3 2013-01-01 10:00:00
##  4 2013-01-01 10:00:00
##  5 2013-01-01 11:00:00
##  6 2013-01-01 10:00:00
##  7 2013-01-01 11:00:00
##  8 2013-01-01 11:00:00
##  9 2013-01-01 11:00:00
## 10 2013-01-01 11:00:00

We could also call : to generate a contiguous sequence:

sr[1:10,]

## ’flights’ (ndoc:10, nfield:19)
##    year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##  1 2013     1   1      517            515         2      830            819
##  2 2013     1   1      533            529         4      850            830
##  3 2013     1   1      542            540         2      923            850
##  4 2013     1   1      544            545        -1     1004           1022
##  5 2013     1   1      554            600        -6      812            837
##  6 2013     1   1      554            558        -4      740            728
##  7 2013     1   1      555            600        -5      913            854
##  8 2013     1   1      557            600        -3      709            723
##  9 2013     1   1      557            600        -3      838            846
## 10 2013     1   1      558            600        -2      753            745
##    arr_delay carrier flight tailnum origin dest air_time distance hour minute
##  1        11      UA   1545  N14228    EWR  IAH      227     1400    5     15
##  2        20      UA   1714  N24211    LGA  IAH      227     1416    5     29
##  3        33      AA   1141  N619AA    JFK  MIA      160     1089    5     40
##  4       -18      B6    725  N804JB    JFK  BQN      183     1576    5     45
##  5       -25      DL    461  N668DN    LGA  ATL      116      762    6      0
##  6        12      UA   1696  N39463    EWR  ORD      150      719    5     58
##  7        19      B6    507  N516JB    EWR  FLL      158     1065    6      0
##  8       -14      EV   5708  N829AS    LGA  IAD       53      229    6      0
##  9        -8      B6     79  N593JB    JFK  MCO      140      944    6      0
## 10         8      AA    301  N3ALAA    LGA  ORD      138      733    6      0
##              time_hour
##  1 2013-01-01 10:00:00
##  2 2013-01-01 10:00:00
##  3 2013-01-01 10:00:00
##  4 2013-01-01 10:00:00
##  5 2013-01-01 11:00:00
##  6 2013-01-01 10:00:00
##  7 2013-01-01 11:00:00
##  8 2013-01-01 11:00:00
##  9 2013-01-01 11:00:00
## 10 2013-01-01 11:00:00

Unfortunately, it is generally infeasible to randomly access Solr records by index, because numeric indexing is a foreign concept to a search engine. Solr does however support retrieval by a key that has a unique value for each document. These data lack such a key, but it is easy to add one and indicate as such to deriveSolrSchema().

2.4 Sorting

To sort the data, we just call sort() and describe the order by passing a formula via the by argument. For example, we sort by year, breaking ties with month, then day:

sort(sr, by = ~ year + month + day)

## ’flights’ (ndoc:336776, nfield:19)
##        year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##      1 2013     1   1      517            515         2      830            819
##      2 2013     1   1      533            529         4      850            830
##      3 2013     1   1      542            540         2      923            850
##      4 2013     1   1      544            545        -1     1004           1022
##      5 2013     1   1      554            600        -6      812            837
##    ...  ...   ... ...      ...            ...       ...      ...            ...
## 336772 2013    12  31     <NA>            705      <NA>     <NA>            931
## 336773 2013    12  31     <NA>            825      <NA>     <NA>           1029
## 336774 2013    12  31     <NA>           1615      <NA>     <NA>           1800
## 336775 2013    12  31     <NA>            600      <NA>     <NA>            735
## 336776 2013    12  31     <NA>            830      <NA>     <NA>           1154
##        arr_delay carrier flight tailnum origin dest air_time distance hour
##      1        11      UA   1545  N14228    EWR  IAH      227     1400    5
##      2        20      UA   1714  N24211    LGA  IAH      227     1416    5
##      3        33      AA   1141  N619AA    JFK  MIA      160     1089    5
##      4       -18      B6    725  N804JB    JFK  BQN      183     1576    5
##      5       -25      DL    461  N668DN    LGA  ATL      116      762    6
##    ...       ...     ...    ...     ...    ...  ...      ...      ...  ...
## 336772      <NA>      UA   1729    <NA>    EWR  DEN     <NA>     1605    7
## 336773      <NA>      US   1831    <NA>    JFK  CLT     <NA>      541    8
## 336774      <NA>      MQ   3301  N844MQ    LGA  RDU     <NA>      431   16
## 336775      <NA>      UA    219    <NA>    EWR  ORD     <NA>      719    6
## 336776      <NA>      UA    443    <NA>    JFK  LAX     <NA>     2475    8
##        minute           time_hour
##      1     15 2013-01-01 10:00:00
##      2     29 2013-01-01 10:00:00
##      3     40 2013-01-01 10:00:00
##      4     45 2013-01-01 10:00:00
##      5      0 2013-01-01 11:00:00
##    ...    ...                 ...
## 336772      5 2013-12-31 12:00:00
## 336773     25 2013-12-31 13:00:00
## 336774     15 2013-12-31 21:00:00
## 336775      0 2013-12-31 11:00:00
## 336776     30 2013-12-31 13:00:00

To sort in decreasing order, just pass decreasing=TRUE as usual:

sort(sr, by = ~ arr_delay, decreasing=TRUE)

## ’flights’ (ndoc:336776, nfield:19)
##        year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##      1 2013     1   9      641            900      1301     1242           1530
##      2 2013     6  15     1432           1935      1137     1607           2120
##      3 2013     1  10     1121           1635      1126     1239           1810
##      4 2013     9  20     1139           1845      1014     1457           2210
##      5 2013     7  22      845           1600      1005     1044           1815
##    ...  ...   ... ...      ...            ...       ...      ...            ...
## 336772 2013     5   4     1816           1820        -4     2017           2131
## 336773 2013     5   2     1947           1949        -2     2209           2324
## 336774 2013     5   6     1826           1830        -4     2045           2200
## 336775 2013     5  20      719            735       -16      951           1110
## 336776 2013     5   7     1715           1729       -14     1944           2110
##        arr_delay carrier flight tailnum origin dest air_time distance hour
##      1      1272      HA     51  N384HA    JFK  HNL      640     4983    9
##      2      1127      MQ   3535  N504MQ    JFK  CMH       74      483   19
##      3      1109      MQ   3695  N517MQ    EWR  ORD      111      719   16
##      4      1007      AA    177  N338AA    JFK  SFO      354     2586   18
##      5       989      MQ   3075  N665MQ    JFK  CVG       96      589   16
##    ...       ...     ...    ...     ...    ...  ...      ...      ...  ...
## 336772       -74      AS      7  N551AS    EWR  SEA      281     2402   18
## 336773       -75      UA    612  N851UA    EWR  LAX      300     2454   19
## 336774       -75      AA    269  N3KCAA    JFK  SEA      289     2422   18
## 336775       -79      VX     11  N840VA    JFK  SFO      316     2586    7
## 336776       -86      VX    193  N843VA    EWR  SFO      315     2565   17
##        minute           time_hour
##      1      0 2013-01-09 14:00:00
##      2     35 2013-06-15 23:00:00
##      3     35 2013-01-10 21:00:00
##      4     45 2013-09-20 22:00:00
##      5      0 2013-07-22 20:00:00
##    ...    ...                 ...
## 336772     20 2013-05-04 22:00:00
## 336773     49 2013-05-02 23:00:00
## 336774     30 2013-05-06 22:00:00
## 336775     35 2013-05-20 11:00:00
## 336776     29 2013-05-07 21:00:00

2.5 Restricting by field

Just as we can use subset to restrict by row, we can also use it to restrict by column:

subset(sr, select=c(year, month, day))

## ’flights’ (ndoc:336776, nfield:3)
##        year month day
##      1 2013     1   1
##      2 2013     1   1
##      3 2013     1   1
##      4 2013     1   1
##      5 2013     1   1
##    ...  ...   ... ...
## 336772 2013     9  30
## 336773 2013     9  30
## 336774 2013     9  30
## 336775 2013     9  30
## 336776 2013     9  30

The select argument is analogous to that of subset.data.frame: it is evaluated to set of field names to which the dataset is restricted. The above example is static, so it is equivalent to:

sr[c("year", "month", "day")]

## ’flights’ (ndoc:336776, nfield:3)
##        year month day
##      1 2013     1   1
##      2 2013     1   1
##      3 2013     1   1
##      4 2013     1   1
##      5 2013     1   1
##    ...  ...   ... ...
## 336772 2013     9  30
## 336773 2013     9  30
## 336774 2013     9  30
## 336775 2013     9  30
## 336776 2013     9  30

But with subset we can also specify dynamic expressions, including ranges:

subset(sr, select=year:day)

## ’flights’ (ndoc:336776, nfield:3)
##        year month day
##      1 2013     1   1
##      2 2013     1   1
##      3 2013     1   1
##      4 2013     1   1
##      5 2013     1   1
##    ...  ...   ... ...
## 336772 2013     9  30
## 336773 2013     9  30
## 336774 2013     9  30
## 336775 2013     9  30
## 336776 2013     9  30

And exclusion:

subset(sr, select=-(year:day))

## ’flights’ (ndoc:336776, nfield:16)
##        dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay
##      1      517            515         2      830            819        11
##      2      533            529         4      850            830        20
##      3      542            540         2      923            850        33
##      4      544            545        -1     1004           1022       -18
##      5      554            600        -6      812            837       -25
##    ...      ...            ...       ...      ...            ...       ...
## 336772     <NA>           1455      <NA>     <NA>           1634      <NA>
## 336773     <NA>           2200      <NA>     <NA>           2312      <NA>
## 336774     <NA>           1210      <NA>     <NA>           1330      <NA>
## 336775     <NA>           1159      <NA>     <NA>           1344      <NA>
## 336776     <NA>            840      <NA>     <NA>           1020      <NA>
##        carrier flight tailnum origin dest air_time distance hour minute
##      1      UA   1545  N14228    EWR  IAH      227     1400    5     15
##      2      UA   1714  N24211    LGA  IAH      227     1416    5     29
##      3      AA   1141  N619AA    JFK  MIA      160     1089    5     40
##      4      B6    725  N804JB    JFK  BQN      183     1576    5     45
##      5      DL    461  N668DN    LGA  ATL      116      762    6      0
##    ...     ...    ...     ...    ...  ...      ...      ...  ...    ...
## 336772      9E   3393    <NA>    JFK  DCA     <NA>      213   14     55
## 336773      9E   3525    <NA>    LGA  SYR     <NA>      198   22      0
## 336774      MQ   3461  N535MQ    LGA  BNA     <NA>      764   12     10
## 336775      MQ   3572  N511MQ    LGA  CLE     <NA>      419   11     59
## 336776      MQ   3531  N839MQ    LGA  RDU     <NA>      431    8     40
##                  time_hour
##      1 2013-01-01 10:00:00
##      2 2013-01-01 10:00:00
##      3 2013-01-01 10:00:00
##      4 2013-01-01 10:00:00
##      5 2013-01-01 11:00:00
##    ...                 ...
## 336772 2013-09-30 18:00:00
## 336773 2013-10-01 02:00:00
## 336774 2013-09-30 16:00:00
## 336775 2013-09-30 15:00:00
## 336776 2013-09-30 12:00:00

Solr also has native support for globs:

sr[c("arr_*", "dep_*")]

## ’flights’ (ndoc:336776, nfield:4)
##        arr_time arr_delay dep_time dep_delay
##      1      830        11      517         2
##      2      850        20      533         4
##      3      923        33      542         2
##      4     1004       -18      544        -1
##      5      812       -25      554        -6
##    ...      ...       ...      ...       ...
## 336772     <NA>      <NA>     <NA>      <NA>
## 336773     <NA>      <NA>     <NA>      <NA>
## 336774     <NA>      <NA>     <NA>      <NA>
## 336775     <NA>      <NA>     <NA>      <NA>
## 336776     <NA>      <NA>     <NA>      <NA>

While we are dealing with fields, we should mention that renaming is also (in principle) possible:

### FIXME: broken in current Solr CSV writer
### rename(sr, tail_num = "tailnum")

2.6 Transformation

To compute new columns from existing ones, we can, as usual, call the transform function:

sr2 <- transform(sr,
                 gain = arr_delay - dep_delay,
                 speed = distance / air_time * 60)
sr2[c("gain", "speed")]

## ’flights’ (ndoc:336776, nfield:1)
##        gain
##      1    9
##      2   16
##      3   31
##      4  -17
##      5  -19
##    ...  ...
## 336772 <NA>
## 336773 <NA>
## 336774 <NA>
## 336775 <NA>
## 336776 <NA>

2.6.1 Advanced note

The transform function essentially quotes and evaluates its arguments in the given frame, and then adds the results as columns in the return value. Direct evaluation affords more flexibility, such as constructing a table with only the newly computed columns. By default, evaluation is completely eager — each referenced column is downloaded in its entirety. But we can make the computation lazier by calling defer prior to the evaluation via with:

with(defer(sr), data.frame(gain = head(arr_delay - dep_delay),
                           speed = head(distance / air_time * 60)))

##   gain    speed
## 1    9 370.0440
## 2   16 374.2731
## 3   31 408.3750
## 4  -17 516.7213
## 5  -19 394.1379
## 6   16 287.6000

Note that this approach, even though it is partially deferred, is potentially less efficient than transform two reasons:

  1. 1.

    It makes two requests to the database, one for each column,

  2. 2.

    The two result columns are downloaded eagerly, since the result must be a data.frame (and thus practicalities required us to take the head of each promised column prior to constructing the data frame).

We can work around the second limitation by using a more general form of data frame, the DataFrame object from S4Vectors:

with(defer(sr),
     S4Vectors::DataFrame(gain = arr_delay - dep_delay,
                          speed = distance / air_time * 60))

## DataFrame with 336776 rows and 2 columns
##                         gain                 speed
##        <SolrFunctionPromise> <SolrFunctionPromise>
## 1                          9             370.04404
## 2                         16             374.27313
## 3                         31               408.375
## 4                        -17              516.7213
## 5                        -19             394.13794
## ...                      ...                   ...
## 336772                    NA                    NA
## 336773                    NA                    NA
## 336774                    NA                    NA
## 336775                    NA                    NA
## 336776                    NA                    NA

Note that we did not need to take the head of the individual columns, since DataFrame does not require the data to be stored in-memory as a base R vector.

2.7 Summarization

Data summarization is about reducing large, complex data to smaller, simpler data that we can understand.

A common type of summarization is aggregation, which is typically defined as a three step process:

  1. 1.

    Split the data into groups, usually by the the interaction of some factor set,

  2. 2.

    Summarize each group to a single value,

  3. 3.

    Combine the summaries.

Solr natively supports the following types of data aggregation:

  • •

    mean,

  • •

    min, max,

  • •

    median, quantile,

  • •

    var, sd,

  • •

    sum,

  • •

    count (table),

  • •

    counting of unique values (for which we introduce nunique).

The rsolr package combines and modifies these operations to support high-level summaries corresponding to the R functions any, all, range, weighted.mean, IQR, mad, etc.

A prerequisite of aggregation is finding the distinct field combinations that correspond to each correspond to a group. Those combinations themselves constitute a useful summary, and we can retrieve them with unique:

unique(sr["tailnum"])

## DocDataFrame (4044x1)
##      tailnum
##    1  D942DN
##    2  N0EGMQ
##    3  N10156
##    4  N102UW
##    5  N103US
##  ...     ...
## 4040  N998AT
## 4041  N998DL
## 4042  N999DN
## 4043  N9EAMQ
## 4044    <NA>
unique(sr[c("origin", "tailnum")])
## DocDataFrame (7944x2)
##      origin tailnum
##    1    EWR  N0EGMQ
##    2    EWR  N10156
##    3    EWR  N102UW
##    4    EWR  N103US
##    5    EWR  N104UW
##  ...    ...     ...
## 7940    LGA  N998AT
## 7941    LGA  N998DL
## 7942    LGA  N999DN
## 7943    LGA  N9EAMQ
## 7944    LGA    <NA>

Solr also supports extracting the top or bottom N documents, after ranking by some field, optionally by group.

The convenient, top-level function for aggregating data is aggregate. To compute a global aggregation, we just specify the computation as an expression (via a named argument, mimicking transform):

aggregate(sr, delay = mean(dep_delay, na.rm=TRUE))

##      delay
## 1 12.63907

It is also possible to specify a function (as the FUN argument), which would be passed the entire frame.

As with stats::aggregate, we can pass a grouping as a formula:

delay <- aggregate(~ tailnum, sr,
                       count = TRUE,
                       dist = mean(distance, na.rm=TRUE),
                       delay = mean(arr_delay, na.rm=TRUE))
delay <- subset(delay, count > 20 & dist < 2000)

The special count argument is a convenience for the common case of computing the number of documents in each group.

Here is an example of using nunique and ndoc:

head(aggregate(~ dest, sr,
               nplanes = nunique(tailnum),
               nflights = ndoc(tailnum)))

##   dest nplanes nflights
## 1  ABQ     108      254
## 2  ACK      58      265
## 3  ALB     172      439
## 4  ANC       6        8
## 5  ATL    1180    17215
## 6  AUS     993     2439

There is limited support for dynamic expressions in the aggregation formula. At a minimum, the expression should evaluate to logical. For example, we can condition on whether the distance is more than 1000 miles.

head(aggregate(~ I(distance > 1000) + tailnum, sr,
               delay = mean(arr_delay, na.rm=TRUE)))

##   I(distance > 1000) tailnum     delay
## 1              FALSE  D942DN 31.500000
## 2              FALSE  N0EGMQ  8.986755
## 3              FALSE  N10156 13.701149
## 4              FALSE  N102UW  2.937500
## 5              FALSE  N103US -6.934783
## 6              FALSE  N104UW  1.804348

It also works for values naturally coercible to logical, such as using the modulus to identify odd numbers. For clarity, we label the variable using transform prior to aggregating.

head(aggregate(~ odd + tailnum, transform(sr, odd = distance %% 2),
               delay = mean(arr_delay, na.rm=TRUE)))

##     odd tailnum     delay
## 1 FALSE  D942DN 31.500000
## 2 FALSE  N0EGMQ  8.589520
## 3 FALSE  N10156  7.797753
## 4 FALSE  N102UW 19.000000
## 5 FALSE  N103US -7.285714
## 6 FALSE  N104UW 20.700000

Aggregate and subset in the same command, as with data.frame:

head(aggregate(~ tailnum, sr,
               subset = distance > 500,
               delay = mean(arr_delay, na.rm=TRUE)))

##   tailnum     delay
## 1  D942DN 31.500000
## 2  N0EGMQ  8.919580
## 3  N10156 12.009174
## 4  N102UW  2.937500
## 5  N103US -6.934783
## 6  N104UW  1.804348

Aggregate the entire dataset:

aggregate(sr, delay = mean(arr_delay, na.rm=TRUE))

##      delay
## 1 6.895377

3 Cleaning up

Having finished our demonstration, we kill our Solr server:

solr$kill()

## Warning in getMethods(coerce, table = TRUE): ’getMethods’ is deprecated.
## Use ’getMethodsForDispatch(f, TRUE)’ instead.
## See help("Deprecated")