Package 'ff'

Description

Yet another assignment interface in order to allow to formulate x[index,...,add=TRUE]<-value in a way which works transparently, not only for ff, but also for ram objects: add(x, value, index, ...).

Usage

add(x, ...)
## S3 method for class 'ff'
add(x, value, ...)
## Default S3 method:
add(x, value, ...)

Arguments

Value

invisible()

Note

Note that add.default changes the object in its parent frame and thus violates R's usual functional programming logic. Duplicated index positions should be avoided, because ff and ram objects behave differently:

  add.ff(x, 1, c(3,3))
  # will increment x at position 3 TWICE by 1, while
  add.default(x, 1, c(3,3))
  # will increment x at position 3 just ONCE by 1
  

Author(s)

Jens Oehlschlägel

See Also

swap, [.ff, LimWarn

Examples

message("incrementing parts of a vector")
   x <- ff(0, length=12)
   y <- rep(0, 12)
   add(x, 1, 1:6)
   add(y, 1, 1:6)
   x
   y

   message("incrementing parts of a matrix")
   x <- ff(0, dim=3:4)
   y <- array(0, dim=3:4)
   add(x, 1, 1:2, 1:2)
   add(y, 1, 1:2, 1:2)
   x
   y

   message("BEWARE that ff and ram methods differ in treatment of duplicated index positions")
   add(x, 1, c(3,3))
   add(y, 1, c(3,3))
   x
   y

   rm(x); gc()

Description

Makes a vector from an array respecting dim and dimorder

Usage

array2vector(x, dim = NULL, dimorder = NULL)

Arguments

Details

This is the inverse function of vector2array. It extracts the vector from the array by first moving through the fastest rotating dimension dim[dimorder[1]], then dim[dimorder[2]], and so forth

Value

a vector

Author(s)

Jens Oehlschlägel

See Also

vector2array, arrayIndex2vectorIndex

Examples

array2vector(matrix(1:12, 3, 4))
  array2vector(matrix(1:12, 3, 4, byrow=TRUE), dimorder=2:1)

Description

Make vector positions from a (non-symmetric) array index respecting dim and dimorder

Usage

arrayIndex2vectorIndex(x, dim = NULL, dimorder = NULL, vw = NULL)

Arguments

Details

The fastest rotating dimension is dim[dimorder[1]], then dim[dimorder[2]], and so forth.
The parameters 'x' and 'dim' may refer to a subarray of a larger array, in this case, the array indices 'x' are interpreted as 'vw[1,] + x' within the larger array 'as.integer(colSums(vw))'.

Value

a vector of indices in seq_len(prod(dim)) (or seq_len(prod(colSums(vw))))

Author(s)

Jens Oehlschlägel

See Also

array2vector, vectorIndex2arrayIndex

Examples

x <- matrix(1:12, 3, 4)
  x
  arrayIndex2vectorIndex(cbind(as.vector(row(x)), as.vector(col(x)))
  , dim=dim(x))
  arrayIndex2vectorIndex(cbind(as.vector(row(x)), as.vector(col(x)))
  , dim=dim(x), dimorder=2:1)
  matrix(1:30, 5, 6)
  arrayIndex2vectorIndex(cbind(as.vector(row(x)), as.vector(col(x)))
  , vw=rbind(c(0,1), c(3,4), c(2,1)))
  arrayIndex2vectorIndex(cbind(as.vector(row(x)), as.vector(col(x)))
  , vw=rbind(c(0,1), c(3,4), c(2,1)), dimorder=2:1)

Description

Coercing ram to ff and ff to ram objects while optionally modifying object features.

Usage

as.ff(x, ...)
  as.ram(x, ...)
  ## Default S3 method:
as.ff(x, filename = NULL, overwrite = FALSE, ...)
  ## S3 method for class 'ff'
as.ff(x, filename = NULL, overwrite = FALSE, ...)
  ## Default S3 method:
as.ram(x, ...)
  ## S3 method for class 'ff'
as.ram(x, ...)

Arguments

Details

If as.ff.ff is called on an 'ff' object or as.ram.default is called on a non-ff object AND no changes are required, the input object 'x' is returned unchanged. Otherwise the workhorse clone.ff is called. If no change of features are requested, the filename attached to the object remains unchanged, otherwise a new filename is requested (or can be set by the user).

Value

A ram or ff object.

Note

If you use ram <- as.ram(ff) for caching, please note that you must close.ff before you can write back as.ff(ram, overwrite=TRUE) (see examples).

Author(s)

Jens Oehlschlägel

See Also

as.ff.bit, ff, clone , as.vmode, vmode, as.hi

Examples

message("create ff")
   myintff <- ff(1:12)
   message("coerce (=clone) integer ff to double ff")
   mydoubleff <- as.ff(myintff, vmode="double")
   message("cache (=clone) integer ff to integer ram AND close original ff")
   myintram <- as.ram(myintff) # filename is retained
   close(myintff)
   message("modify ram cache and write back (=clone) to ff")
   myintram[1] <- -1L
   myintff <- as.ff(myintram, overwrite=TRUE)
   message("coerce (=clone) integer ram to double ram")
   mydoubleram <- as.ram(myintram, vmode="double")
   message("coerce (inplace) integer ram to double ram")
   myintram <- as.ram(myintram, vmode="double")
   message("more classic: coerce (inplace) double ram to integer ram")
   vmode(myintram) <- "integer"
   rm(myintff, myintram, mydoubleff, mydoubleram); gc()

Description

Function as.ff.bit converts a bit vector to a boolean ff vector. Function as.bit.ff converts a boolean ff vector to a ff vector.

Usage

## S3 method for class 'bit'
as.ff(x, filename = NULL, overwrite = FALSE, ...)
## S3 method for class 'ff'
as.bit(x, ...)

Arguments

Details

The data are copied bot bit-wise but integerwise, therefore these conversions are very fast. as.bit.ff will attach the ff filename to the bit vector, and as.ff.bit will - if attached - use THIS filename and SILENTLY overwrite this file.

Value

A vector of the converted type

Note

NAs are mapped to TRUE in 'bit' and to FALSE in 'ff' booleans. Might be aligned in a future release. Don't use bit if you have NAs - or map NAs explicitely.

Author(s)

Jens Oehlschlägel

See Also

bit, ff, as.ff, as.hi.bit

Examples

l <- as.boolean(sample(c(FALSE,TRUE), 1000, TRUE))

  b <- as.bit(l)
  stopifnot(identical(l,b[]))
  b
  f <- as.ff(b)
  stopifnot(identical(l,f[]))
  f
  b2 <- as.bit(f)
  stopifnot(identical(l,b2[]))
  b2
  f2 <- as.ff(b2)
  stopifnot(identical(filename(f),filename(f2)))
  stopifnot(identical(l,f2[]))
  f
  rm(f,f2); gc()

Description

Functions for coercing to ffdf and data.frame

Usage

as.ffdf(x, ...)
## S3 method for class 'ff_vector'
as.ffdf(x, ...)
## S3 method for class 'ff_matrix'
as.ffdf(x, ...)
## S3 method for class 'data.frame'
as.ffdf(x, vmode=NULL, col_args = list(), ...)
## S3 method for class 'ffdf'
as.data.frame(x, ...)

Arguments

Value

'as.ffdf' returns an object of class ffdf, 'as.data.frame' returns an object of class data.frame

Author(s)

Jens Oehlschlägel

See Also

is.ffdf, ffdf, data.frame

Examples

d <- data.frame(x=1:26, y=letters, z=Sys.time()+1:26, stringsAsFactors = TRUE)
  ffd <- as.ffdf(d)
  stopifnot(identical(d, as.data.frame(ffd)))
  rm(ffd); gc()

Description

The generic as.hi and its methods are the main (internal) means for preprocessing index information into the hybrid index class hi. Usually as.hi is called transparently from [.ff. However, you can explicitely do the index-preprocessing, store the Hybrid Index hi, and use the hi for subscripting.

Usage

as.hi(x, ...)
## S3 method for class 'NULL'
as.hi(x, ...)
## S3 method for class 'hi'
as.hi(x, ...)
## S3 method for class 'ri'
as.hi(x, maxindex = length(x), ...)
## S3 method for class 'bit'
as.hi(x, range = NULL, maxindex = length(x), vw = NULL
, dim = NULL, dimorder = NULL, pack = TRUE, ...)
## S3 method for class 'bitwhich'
as.hi(x, maxindex = length(x), pack = FALSE, ...)
## S3 method for class 'call'
as.hi(x, maxindex = NA, dim = NULL, dimorder = NULL, vw = NULL
, vw.convert = TRUE, pack = TRUE, envir = parent.frame(), ...)
## S3 method for class 'name'
as.hi(x, envir = parent.frame(), ...)
## S3 method for class 'integer'
as.hi(x, maxindex = NA, dim = NULL, dimorder = NULL
, symmetric = FALSE, fixdiag = NULL, vw = NULL, vw.convert = TRUE
, dimorder.convert  = TRUE, pack = TRUE, NAs = NULL, ...)
## S3 method for class 'which'
as.hi(x, ...)
## S3 method for class 'double'
as.hi(x, ...)
## S3 method for class 'logical'
as.hi(x, maxindex = NA, dim = NULL, vw = NULL, pack = TRUE, ...)
## S3 method for class 'character'
as.hi(x, names, vw = NULL, vw.convert = TRUE, ...)
## S3 method for class 'matrix'
as.hi(x, dim, dimorder = NULL, symmetric = FALSE, fixdiag = NULL
, vw = NULL, pack = TRUE, ...)

Arguments

Details

The generic dispatches appropriately, as.hi.hi returns an hi object unchanged, as.hi.call tries to hiparse instead of evaluate its input in order to save RAM. If parsing is successfull as.hi.call will ignore its argument pack and always pack unless the subscript is too small to do so. If parsing fails it evaluates the index expression and dispatches again to one of the other methods. as.hi.name and as.hi.( are wrappers to as.hi.call. as.hi.integer is the workhorse for coercing evaluated expressions and as.hi.which is a wrapper removing the which class attribute. as.hi.double, as.hi.logical and as.hi.character are also wrappers to as.hi.integer, but note that as.hi.logical is not memory efficient because it expands all positions and then applies logical subscripting.
as.hi.matrix calls arrayIndex2vectorIndex and then as.hi.integer to interpret and preprocess matrix indices.
If the dim and dimorder parameter indicate a non-standard dimorder (dimorderStandard), the index information in x is converted from a standard dimorder interpretation to the requested dimorder.
If the vw parameter is used, the index information in x is interpreted relative to the virtual window but stored relative to the abolute origin. Back-coercion via as.integer.hi and friends will again return the index information relative to the virtual window, thus retaining symmetry and transparency of the viurtual window to the user.
You can use length to query the index length (possibly length of negative subscripts), poslength to query the number of selected elements (even with negative subscripts), and maxindex to query the largest possible index position (within virtual window, if present)
Duplicated negative indices are removed and will not be recovered by as.integer.hi.

Value

an object of class hi

Note

Avoid changing the Hybrid Index representation, this might crash the [.ff subscripting.

Author(s)

Jens Oehlschlägel

See Also

hi for the Hybrid Index class, hiparse for parsing details, as.integer.hi for back-coercion, [.ff for ff subscripting

Examples

message("integer indexing with and without rel-packing")
  as.hi(1:12)
  as.hi(1:12, pack=FALSE)
  message("if index is double, the wrapper method just converts to integer")
  as.hi(as.double(1:12))
  message("if index is character, the wrapper method just converts to integer")
  as.hi(c("a","b","c"), names=letters)
  message("negative index must use maxindex (or vw)")
  as.hi(-(1:3), maxindex=12)
  message("logical index can use maxindex")
  as.hi(c(FALSE, FALSE, TRUE, TRUE))
  as.hi(c(FALSE, FALSE, TRUE, TRUE), maxindex=12)

  message("matrix index")
  x <- matrix(1:12, 6)
  as.hi(rbind(c(1,1), c(1,2), c(2,1)), dim=dim(x))

  message("first ten positions within virtual window")
  i <- as.hi(1:10, vw=c(10, 80, 10))
  i
  message("back-coerce relativ to virtual window")
  as.integer(i)
  message("back-coerce relativ to absolute origin")
  as.integer(i, vw.convert=FALSE)

  message("parsed index expressions save index RAM")
    as.hi(quote(1:1000000000))
## Not run: 
  message("compare to RAM requirement when the index experssion is evaluated")
    as.hi(1:1000000000)

## End(Not run)

message("example of parsable index expression")
  a <- seq(100, 200, 20)
  as.hi(substitute(c(1:5, 4:9, a)))
  hi(c(1,4, 100),c(5,9, 200), by=c(1,1,20))

message("two examples of index expression temporarily expanded to full length due to 
non-supported use of brackets '(' and mathematical operators '+' accepting token")
message("example1: accepted token but aborted parsing because length>16")
  as.hi(quote(1+(1:16)))
message("example1: rejected token and aborted parsing because length>16")
  as.hi(quote(1+(1:17)))

Description

Functions that (back-)convert an hi object to the respective subscripting information.

Usage

## S3 method for class 'hi'
as.which(x, ...)
## S3 method for class 'hi'
as.bitwhich(x, ...)
## S3 method for class 'hi'
as.bit(x, ...)
## S3 method for class 'hi'
as.integer(x, vw.convert = TRUE, ...)
## S3 method for class 'hi'
as.logical(x, maxindex = NULL, ...)
## S3 method for class 'hi'
as.character(x, names, vw.convert = TRUE, ...)
## S3 method for class 'hi'
as.matrix(x, dim = x$dim, dimorder = x$dimorder
, vw = x$vw, symmetric = x$symmetric, fixdiag = x$fixdiag, ...)

Arguments

Value

as.integer.hi returns an integer vector, see as.hi.integer. as.logical.hi returns an logical vector, see as.hi.logical. as.character.hi returns a character vector, see as.hi.character. as.matrix.hi returns a matrix index, see as.hi.matrix.

Author(s)

Jens Oehlschlägel

See Also

hi, as.hi

Examples

x <- 1:6
  names(x) <- letters[1:6]
  as.integer(as.hi(c(1:3)))
  as.logical(as.hi(c(TRUE,TRUE,TRUE,FALSE,FALSE,FALSE)))
  as.character(as.hi(letters[1:3], names=names(x)), names=names(x))
  x <- matrix(1:12, 6)
  as.matrix(as.hi(rbind(c(1,1), c(1,2), c(2,1)), dim=dim(x)), dim=dim(x))

Description

as.vmode is a generic that converts some R ram object to the desired vmode.

Usage

as.vmode(x, ...)
as.boolean(x, ...)
as.quad(x, ...)
as.nibble(x, ...)
as.byte(x, ...)
as.ubyte(x, ...)
as.short(x, ...)
as.ushort(x, ...)
## Default S3 method:
as.vmode(x, vmode, ...)
## S3 method for class 'ff'
as.vmode(x, ...)
## Default S3 method:
as.boolean(x, ...)
## Default S3 method:
as.quad(x, ...)
## Default S3 method:
as.nibble(x, ...)
## Default S3 method:
as.byte(x, ...)
## Default S3 method:
as.ubyte(x, ...)
## Default S3 method:
as.short(x, ...)
## Default S3 method:
as.ushort(x, ...)

Arguments

Details

Function as.vmode actually coerces to one of the usual storage.modes (see .rammode) but flags them with an additional attribute 'vmode' if necessary. The coercion generics can also be called directly:

Value

a vector of the desired vmode containing the input data

Author(s)

Jens Oehlschlägel

See Also

vmode, vector.vmode

Examples

as.vmode(1:3,"double")
  as.vmode(1:3,"byte")
  as.double(1:3)
  as.byte(1:3)

Description

bigsample samples quicker from large pools than sample does.

Usage

bigsample(x, ...)
## Default S3 method:
bigsample(x, size, replace = FALSE, prob = NULL, negative = FALSE, ...)
## S3 method for class 'ff'
bigsample(x, size, replace = FALSE, prob = NULL, ...)

Arguments

Details

For small pools sample is called.

Value

a vector of elements sampled from the pool (argument 'x')

Note

Note that bigsample and sample do not necessarily return the same sequence of elements when set.seed is set before.

Author(s)

Daniel Adler, Jens Oehlschlägel, Walter Zucchini

See Also

sample, ff

Examples

message("Specify pool size")
bigsample(1e8, 10)
message("Sample ff elements (same as x[bigsample(length(ff(1:100 / 10)), 10)])")
bigsample(ff(1:100 / 10), 10)
 ## Not run: 
   message("Speed factor")
     (system.time(for(i in 1:10)sample(1e8, 10))[3]/10) 
   / (system.time(for(i in 1:1000)bigsample(1e8, 10))[3]/1000)
 
## End(Not run)

Description

These are used in aggregating the chunks resulting from batch processing. They are usually called via do.call

Usage

ccbind(...)
crbind(...)
cfun(..., FUN, FUNARGS = list())
cquantile(..., probs = seq(0, 1, 0.25), na.rm = FALSE, names = TRUE, type = 7)
csummary(..., na.rm = "ignored")
cmedian(..., na.rm = FALSE)
clength(..., na.rm = FALSE)
csum(..., na.rm = FALSE)
cmean(..., na.rm = FALSE)

Arguments

Details

In order to use CFUNs on the result of lapply or ffapply use do.call.

Value

depends on the CFUN used

ff options

xx TODO: extend this for weighted means, weighted median etc.,
google "Re: [R] Weighted median"

Note

Currently - for command line convenience - we map the elements of a single list argument to ..., but this may change in the future.

Author(s)

Jens Oehlschlägel

See Also

ffapply, do.call, na.count

Examples

X <- lapply(split(rnorm(1000), 1:10), summary)
   do.call("crbind", X)
   do.call("csummary", X)
   do.call("cmean", X)
   do.call("cfun", c(X, list(FUN=mean, FUNARGS=list(na.rm=TRUE))))
   rm(X)

Description

Chunking method for ff_vector and ffdf objects (row-wise) automatically considering RAM requirements from recordsize as calculated from sum(.rambytes[vmode])

Usage

## S3 method for class 'ff_vector'
chunk(x
, RECORDBYTES = .rambytes[vmode(x)], BATCHBYTES = getOption("ffbatchbytes"), ...)
## S3 method for class 'ffdf'
chunk(x
, RECORDBYTES = sum(.rambytes[vmode(x)]), BATCHBYTES = getOption("ffbatchbytes"), ...)

Arguments

Value

A list with ri indexes each representing one chunk

Author(s)

Jens Oehlschlägel

See Also

chunk, ffdf

Examples

x <- data.frame(x=as.double(1:26), y=factor(letters), z=ordered(LETTERS), stringsAsFactors = TRUE)
  a <- as.ffdf(x)
  ceiling(26 / (300 %/% sum(.rambytes[vmode(a)])))
  chunk(a, BATCHBYTES=300)
  ceiling(13 / (100 %/% sum(.rambytes[vmode(a)])))
  chunk(a, from=1, to = 13, BATCHBYTES=100)
  rm(a); gc()

  message("dummy example for linear regression with biglm on ffdf")
  library(biglm)

  message("NOTE that . in formula requires calculating terms manually
    because . as a data-dependant term is not allowed in biglm")
  form <- Sepal.Length ~ Sepal.Width + Petal.Length + Petal.Width + Species

  lmfit <- lm(form, data=iris)

  firis <- as.ffdf(iris)
  for (i in chunk(firis, by=50)){
    if (i[1]==1){
      message("first chunk is: ", i[[1]],":",i[[2]])
      biglmfit <- biglm(form, data=firis[i,,drop=FALSE])
    }else{
      message("next chunk is: ", i[[1]],":",i[[2]])
      biglmfit <- update(biglmfit, firis[i,,drop=FALSE])
    }
  }

  summary(lmfit)
  summary(biglmfit)
  stopifnot(all.equal(coef(lmfit), coef(biglmfit)))

Description

clone physically duplicates ff (and ram) objects and can additionally change some features, e.g. length.

Usage

## S3 method for class 'ff'
clone(x
, initdata = x
, length = NULL
, levels = NULL
, ordered = NULL
, dim = NULL
, dimorder = NULL
, bydim = NULL
, symmetric = NULL
, fixdiag = NULL
, names = NULL
, dimnames = NULL
, ramclass = NULL
, ramattribs = NULL
, vmode = NULL
, update  = NULL
, pattern = NULL
, filename = NULL
, overwrite = FALSE
, pagesize = NULL
, caching = NULL
, finalizer = NULL
, finonexit = NULL
, FF_RETURN = NULL
, BATCHSIZE = .Machine$integer.max
, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE
, ...)

Arguments

Details

clone is generic. clone.ff is the workhorse behind as.ram and as.ff. For creating the desired object it calls ff which calls update for initialization.

Value

an ff or ram object

Author(s)

Jens Oehlschlägel

See Also

ff, update, as.ram, as.ff

Examples

x <- ff(letters, levels=letters)
  y <- clone(x, length=52)
  rm(x,y); gc()

Description

clone physically duplicates ffdf objects

Usage

## S3 method for class 'ffdf'
clone(x, nrow=NULL, ...)

Arguments

Details

Creates a deep copy of an ffdf object by cloning all physical components including the row.names

Value

An object of type ffdf

Author(s)

Jens Oehlschlägel

See Also

clone, ffdf

Examples

x <- as.ffdf(data.frame(a=1:26, b=letters, stringsAsFactors = TRUE))

  message("Here we change the content of both x and y by reference")
  y <- x
  x$a[1] <- -1
  y$a[1]

  message("Here we change the content only of x because y is a deep copy")
  y <- clone(x)
  x$a[2] <- -2
  y$a[2]
  rm(x, y); gc()

Description

Close frees the Memory Mapping resources and closes the ff file without deleting the file data.

Usage

## S3 method for class 'ff'
close(con, ...)
## S3 method for class 'ffdf'
close(con, ...)
## S3 method for class 'ff_pointer'
close(con, ...)

Arguments

Details

The ff_pointer method is not intended for manual use, it is used at finalizer dispatch time. Closing ffdf objects will close all of their physical components including their row.names if they are is.ff

Value

TRUE if the file could be closed, FALSE if it was closed already (or NA if not all components of an ffdf returned FALSE or TRUE on closing)

Author(s)

Jens Oehlschlägel

See Also

ff, open.ff, delete, deleteIfOpen

Examples

x <- ff(1:12)
  close(x)
  x
  open(x)
  x
  rm(x); gc()

Description

The generic delete deletes the content of an object without removing the object itself. The generic deleteIfOpen does the same, but only if is.open returns TRUE.

Usage

delete(x, ...)
deleteIfOpen(x, ...)
## S3 method for class 'ff'
delete(x, ...)
## S3 method for class 'ffdf'
delete(x, ...)
## S3 method for class 'ff_pointer'
delete(x, ...)
## Default S3 method:
delete(x, ...)
## S3 method for class 'ff'
deleteIfOpen(x, ...)
## S3 method for class 'ff_pointer'
deleteIfOpen(x, ...)

Arguments

Details

The proper sequence to fully delete an ff object is: delete(x);rm(x), where delete.ff frees the Memory Mapping resources and deletes the ff file, leaving intact the R-side object including its class, physical and virtual attributes. The default method is a compatibility function doing something similar with ram objects: by assiging an empty list to the name of the ram object to the parent frame we destroy the content of the object, leaving an empty stub that prevents raising an error if the parent frame calls the delete(x);rm(x) sequence.
The deleteIfOpen does the same as delete but protects closed ff objects from deletion, it is mainly intended for use through a finalizer, as are the ff_pointer methods.

Value

delete returns TRUE if the/all ff files could be removed and FALSE otherwise.
deleteIfOpen returns TRUE if the/all ff files could be removed, FALSE if not and NA if the ff object was open.

Note

Deletion of ff files can be triggerd automatically via three routes:

1. if an ff object with a 'delete' finalizer is removed

2. if an ff object was created with fffinonexit=TRUE the finalizer is also called when R shuts down.

3. if an ff object was created in getOption("fftempdir"), it will be unlinked together with the fftempdir .onUnload

Thus in order to retain an ff file, one has to create it elsewhere than in fftempdir with a finalizer that does not destroy the file (by default files outside fftempdir get a 'close' finalizer') i.e. one of the following:

1. name the file AND use fffinalizer="close"

2. name the file AND use fffinalizer="deleteIfOpen" AND close the ff object before leaving R

3. name the file AND use fffinalizer="delete" AND use fffinonexit=FALSE

Author(s)

Jens Oehlschlägel

See Also

ff, close.ff, open.ff, reg.finalizer

Examples

message('create the ff file outside getOption("fftempir"), 
    it will have default finalizer "close", so you need to delete it explicitely')
  x <- ff(1:12, pattern="./ffexample")
  delete(x)
  rm(x)

Description

Assigning dim to an ff_vector changes it to an ff_array. Beyond that dimorder can be assigned to change from column-major order to row-major order or generalizations for higher order ff_array.

Usage

## S3 method for class 'ff'
dim(x)
  ## S3 method for class 'ffdf'
dim(x)
  ## S3 replacement method for class 'ff'
dim(x) <- value
  ## S3 replacement method for class 'ffdf'
dim(x) <- value
   dimorder(x, ...)
   dimorder(x, ...) <- value
  ## Default S3 method:
dimorder(x, ...)
  ## S3 method for class 'ff_array'
dimorder(x, ...)
  ## S3 method for class 'ffdf'
dimorder(x, ...)
  ## S3 replacement method for class 'ff_array'
dimorder(x, ...) <- value
  ## S3 replacement method for class 'ffdf'
dimorder(x, ...) <- value  # just here to catch forbidden assignments

Arguments

Details

dim and dimorder are virtual attributes. Thus two copies of an R ff object can point to the same file but interpret it differently. dim has the usual meaning, dimorder defines the dimension order of storage, i.e. c(1,2) corresponds to R's standard column-major order, c(1,2) corresponds to row-major order, and for higher dimensional arrays dimorder can also be used. Standard dimorder is seq_along(dim(x)).
For ffdf dim returns the number of rows and virtual columns. With dim<-.ffdf only the number of rows can be changed. For convenience you can assign NA to the number of columns.
For ffdf the dimorder returns non-standard dimorder if any of its columns contains a ff object with non-standard dimorder (see dimorderStandard) An even higher level of virtualization is available using virtual windows, see vw.

Value

names returns a character vector (or NULL)

Note

x[] returns a matrix like x[,] and thus respects dimorder, while x[i:j] returns a vector and simply returns elements in the stored order. Check the corresponding example twice, in order to make sure you understand that for non-standard dimorder x[seq_along(x)] is not the same as as.vector(x[]).

Author(s)

Jens Oehlschlägel

See Also

dim, dimnames.ff_array, dimorderStandard, vw, virtual

Examples

x <- ff(1:12, dim=c(3,4), dimorder=c(2:1))
  y <- x
  dim(y) <- c(4,3)
  dimorder(y) <- c(1:2)
  x
  y
  x[]
  y[]
  x[,bydim=c(2,1)]
  y[,bydim=c(2,1)]

  message("NOTE that x[] like x[,] returns a matrix (respects dimorder),")
  message("while x[1:12] returns a vector IN STORAGE ORDER")
  message("check the following examples twice to make sure you understand this")
  x[,]
  x[]
  as.vector(x[])
  x[1:12]
  rm(x,y); gc()

  
  ## Not run: 
    message("some performance comparison between different dimorders")
    n <- 100
    m <- 100000
    a <- ff(1L,dim=c(n,m))
    b <- ff(1L,dim=c(n,m), dimorder=2:1)
    system.time(lapply(1:n, function(i)sum(a[i,])))
    system.time(lapply(1:n, function(i)sum(b[i,])))
    system.time(lapply(1:n, function(i){i<-(i-1)*(m/n)+1; sum(a[,i:(i+m/n-1)])}))
    system.time(lapply(1:n, function(i){i<-(i-1)*(m/n)+1; sum(b[,i:(i+m/n-1)])}))
    rm(a,b); gc()
  
## End(Not run)

Description

For ff_arrays you can set dimnames.

Usage

## S3 method for class 'ff_array'
dimnames(x)
  ## S3 replacement method for class 'ff_array'
dimnames(x) <- value

Arguments

Details

if vw is set, dimnames.ff_array returns the appropriate part of the names, but you can't set dimnames while vw is set. dimnames returns NULL for ff_vectors and setting dimnames for ff_vector is not allowed, but setting names is.

Value

dimnames returns a list, see dimnames

Author(s)

Jens Oehlschlägel

See Also

dimnames, dim.ff , names.ff , vw, virtual

Examples

x <- ff(1:12, dim=c(3,4), dimnames=list(letters[1:3], LETTERS[1:4]))
  dimnames(x)
  dimnames(x) <- list(LETTERS[1:3], letters[1:4])
  dimnames(x)
  dimnames(x) <- NULL
  dimnames(x)
  rm(x); gc()

Description

Getting and setting dimnames, columnnames or rownames

Usage

## S3 method for class 'ffdf'
dimnames(x)
  ## S3 replacement method for class 'ffdf'
dimnames(x) <- value
  ## S3 method for class 'ffdf'
names(x)
  ## S3 replacement method for class 'ffdf'
names(x) <- value
  ## S3 method for class 'ffdf'
row.names(x)
  ## S3 replacement method for class 'ffdf'
row.names(x) <- value

Arguments

Details

It is recommended not to assign row.names to a large ffdf object.

Value

The assignment function return the changed ffdf object. The other functions return the expected.

Author(s)

Jens Oehlschlägel

See Also

ffdf, dimnames.ff, rownames, colnames

Examples

ffd <- as.ffdf(data.frame(a=1:26, b=letters, stringsAsFactors = TRUE))
  dimnames(ffd)
  row.names(ffd) <- letters
  dimnames(ffd)
  ffd
  rm(ffd); gc()

Description

dimorderStandard returns TRUE if the dimorder is standard (ascending), vectorStandard returns TRUE if the dimorder-bydim combination is compatible with a standard elementwise vector interpretation, dimorderCompatible returns TRUE if two dimorders have a compatible elementwise vector interpretation and vectorCompatible returns TRUE if dimorder-bydim combinations have a compatible elementwise vector interpretation.

Usage

dimorderStandard(dimorder)
vectorStandard(dimorder, bydim = NULL)
dimorderCompatible(dim, dim2, dimorder, dimorder2)
vectorCompatible(dim, dim2, dimorder=NULL, dimorder2=NULL, bydim = NULL, bydim2 = NULL)

Arguments

Value

TRUE if compatibility has been detected, FALSE otherwise

Note

does not yet gurantee to detect all compatible configurations, but the most important ones

Author(s)

Jens Oehlschlägel

See Also

dimorder, ffconform

Description

makes standard dimnames from letters and integers (for testing)

Usage

dummy.dimnames(x)

Arguments

Value

a list with character vectors suitable to be assigned as dimnames to x

Author(s)

Jens Oehlschlägel

See Also

dimnames

Examples

dummy.dimnames(matrix(1:12, 3, 4))

Description

These are the main methods for reading and writing data from ff files.

Usage

## S3 method for class 'ff'
x[i, pack = FALSE]
## S3 replacement method for class 'ff'
x[i, add = FALSE, pack = FALSE] <- value
## S3 method for class 'ff_array'
x[..., bydim = NULL, drop = getOption("ffdrop"), pack = FALSE]
## S3 replacement method for class 'ff_array'
x[..., bydim = NULL, add = FALSE, pack = FALSE] <- value
## S3 method for class 'ff'
x[[i]]
## S3 replacement method for class 'ff'
x[[i, add = FALSE]] <- value

Arguments

Details

The single square bracket operators [ and [<- are the workhorses for accessing the content of an ff object. They support ff_vector and ff_array access (dim.ff), they respect virtual windows (vw), names.ff and dimnames.ff and retain ramclass and ramattribs and thus support POSIXct and factor, see levels.ff.
The functionality of [ and [<- cn be combined into one efficient operation, see swap.
The double square bracket operator [[ is a shortcut for get.ff resp. set.ff, however, you should not rely on this for the future, see LimWarn. For programming please prefer [.

Value

The read operators [ and [[ return data from the ff object, possibly decorated with names, dim, dimnames and further attributes and classes (see ramclass, ramattribs)
The write operators [<- and [[<- return the 'modified' ff object (like all assignment operators do).

Index expressions

x <- ff(1:12, dim=c(3,4), dimnames=list(letters[1:3], NULL))

Dimorder and bydim

Arrays in R have always standard dimorder seq_along(dim(x)) while ff allows to store an array in a different dimorder. Using nonstandard dimorder (see dimorderStandard) can speed up certain access operations: while matrix dimorder=c(1,2) – column-major order – allows fast extraction of columns, dimorder=c(2,1) allows fast extraction of rows.
While the dimorder – being an attribute of an ff_array – controls how the vector in an ff file is interpreted, the bydim argument to the extractor functions controls, how assigment vector values in [<- are translated to the array and how the array is translated to a vector in [ subscripting. Note that bydim=c(2,1) corresponds to matrix(..., byrow=TRUE).

Multiple vector interpretation in arrays

In case of non-standard dimorder (see dimorderStandard) the vector sequence of array elements in R and in the ff file differs. To access array elements in file order, you can use getset.ff, readwrite.ff or copy the ff object and set dim(ff)<-NULL to get a vector view into the ff object (using [ dispatches the vector method [.ff). To access the array elements in R standard dimorder you simply use [ which dispatches to [.ff_array. Note that in this case as.hi will unpack the complete index, see next section.

RAM expansion of index expressions

Some index expressions do not consume RAM due to the hi representation, for example 1:n will almost consume no RAM hoewever large n. However, some index expressions are expanded and require to maxindex(i) * .rambytes["integer"] bytes, either because the sorted sequence of index positions cannot be rle-packed efficiently or because hiparse cannot yet parse such expression and falls back to evaluating/expanding the index expression. If the index positions are not sorted, the index will be expanded and a second vector is needed to store the information for re-ordering, thus the index requires 2 * maxindex(i) * .rambytes["integer"] bytes.

RAM expansion when recycling assigment values

Some assignment expressions do not consume RAM for recycling, for example x[1:n] <- 1:k will not consume RAM hoewever large n compared to k, when x has standard dimorder. However, if length(value)>1, assignment expressions with non-ascending index positions trigger recycling the value R-side to the full index length. This will happen if dimorder does not match parameter bydim or if the index is not sorted ascending.

Author(s)

Jens Oehlschlägel

See Also

ff, swap, add, readwrite.ff, LimWarn

Examples

message("look at different dimorders")
   x <- ff(1:12, dim=c(3,4), dimorder=c(1,2))
   x[]
   as.vector(x[])
   x[1:12]
   x <- ff(1:12, dim=c(3,4), dimorder=c(2,1))
   x[]
   as.vector(x[])
   message("Beware (might be changed)")
   x[1:12]

   message("look at different bydim")
   matrix(1:12, nrow=3, ncol=4, byrow=FALSE)
   x <- ff(1:12, dim=c(3,4), bydim=c(1,2))
   x
   matrix(1:12, nrow=3, ncol=4, byrow=TRUE)
   x <- ff(1:12, dim=c(3,4), bydim=c(2,1))
   x
   x[,, bydim=c(2,1)]
   as.vector(x[,, bydim=c(2,1)])
   message("even consistent interpretation of vectors in assignments")
   x[,, bydim=c(1,2)] <- x[,, bydim=c(1,2)]
   x
   x[,, bydim=c(2,1)] <- x[,, bydim=c(2,1)]
   x
   rm(x); gc()

  ## Not run: 
   message("some performance implications of different dimorders")
   n <- 100
   m <- 100000
   a <- ff(1L,dim=c(n,m))
   b <- ff(1L,dim=c(n,m), dimorder=2:1)
   system.time(lapply(1:n, function(i)sum(a[i,])))
   system.time(lapply(1:n, function(i)sum(b[i,])))
   system.time(lapply(1:n, function(i){i<-(i-1)*(m/n)+1; sum(a[,i:(i+m/n-1)])}))
   system.time(lapply(1:n, function(i){i<-(i-1)*(m/n)+1; sum(b[,i:(i+m/n-1)])}))

   n <- 100
   a <- ff(1L,dim=c(n,n,n,n))
   b <- ff(1L,dim=c(n,n,n,n), dimorder=4:1)
   system.time(lapply(1:n, function(i)sum(a[i,,,])))
   system.time(lapply(1:n, function(i)sum(a[,i,,])))
   system.time(lapply(1:n, function(i)sum(a[,,i,])))
   system.time(lapply(1:n, function(i)sum(a[,,,i])))
   system.time(lapply(1:n, function(i)sum(b[i,,,])))
   system.time(lapply(1:n, function(i)sum(b[,i,,])))
   system.time(lapply(1:n, function(i)sum(b[,,i,])))
   system.time(lapply(1:n, function(i)sum(b[,,,i])))

   n <- 100
   m <- 100000
   a <- ff(1L,dim=c(n,m))
   b <- ff(1L,dim=c(n,m), dimorder=2:1)
   system.time(ffrowapply(sum(a[i1:i2,]), a, RETURN=TRUE, CFUN="csum", BATCHBYTES=16104816%/%20))
   system.time(ffcolapply(sum(a[,i1:i2]), a, RETURN=TRUE, CFUN="csum", BATCHBYTES=16104816%/%20))
   system.time(ffrowapply(sum(b[i1:i2,]), b, RETURN=TRUE, CFUN="csum", BATCHBYTES=16104816%/%20))
   system.time(ffcolapply(sum(b[,i1:i2]), b, RETURN=TRUE, CFUN="csum", BATCHBYTES=16104816%/%20))

   rm(a,b); gc()
  
## End(Not run)

Description

These are the main methods for reading and writing data from ffdf objects.

Usage

## S3 method for class 'ffdf'
x[i, j, drop = ncols == 1]
## S3 replacement method for class 'ffdf'
x[i, j] <- value
## S3 method for class 'ffdf'
x[[i, j, exact = TRUE]]
## S3 replacement method for class 'ffdf'
x[[i, j]] <- value
## S3 method for class 'ffdf'
x$i
## S3 replacement method for class 'ffdf'
x$i <- value

Arguments

Details

The subscript methods [, [[ and $, behave symmetrical to the assignment functions [<-, [[<- and $<-. What the former return is the assignment value to the latter. A notable exception is assigning NULL in [[<- and $<- which removes the virtual column from the ffdf (and the physical component if it is no longer needed by any virtual column). Creating new columns via [[<- and $<- requires giving a name to the new column (character subscripting). [<- does not allow to create new columns, only to replace existing ones.

Subscript expressions and return values

Author(s)

Jens Oehlschlägel

See Also

ffdf, Extract.data.frame, Extract.ff

Examples

d <- data.frame(a=letters, b=rev(letters), c=1:26, stringsAsFactors = TRUE)
   x <- as.ffdf(d)

   d[1,]
   x[1,]

   d[1:2,]
   x[1:2,]

   d[,1]
   x[,1]

   d[,1:2]
   x[,1:2]

   d[cbind(1:2,2:1)]
   x[cbind(1:2,2:1)]

   d[1]
   x[1]

   d[[1]]
   x[[1]]

   d$a
   x$a

   d$a[1:2]
   x$a[1:2]

   rm(x); gc()

Description

The ff package provides atomic data structures that are stored on disk but behave (almost) as if they were in RAM by mapping only a section (pagesize) into main memory (the effective main memory consumption per ff object). Several access optimization techniques such as Hyrid Index Preprocessing (as.hi, update.ff) and Virtualization (virtual, vt, vw) are implemented to achieve good performance even with large datasets. In addition to the basic access functions, the ff package also provides compatibility functions that facilitate writing code for ff and ram objects (clone, as.ff, as.ram) and very basic support for operating on ff objects (ffapply). While the (possibly packed) raw data is stored on a flat file, meta informations about the atomic data structure such as its dimension, virtual storage mode (vmode), factor level encoding, internal length etc.. are stored as an ordinary R object (external pointer plus attributes) and can be saved in the workspace. The raw flat file data encoding is always in native machine format for optimal performance and provides several packing schemes for different data types such as logical, raw, integer and double (in an extended version support for more tighly packed virtual data types is supported). flatfile data files can be shared among ff objects in the same R process or even from different R processes due to Memory-Mapping, although the caching effects have not been tested extensively.
Please do read and understand the limitations and warnings in LimWarn before you do anything serious with package ff.

Usage

ff( initdata  = NULL
, length      = NULL
, levels      = NULL
, ordered     = NULL
, dim         = NULL
, dimorder    = NULL
, bydim       = NULL
, symmetric   = FALSE
, fixdiag     = NULL
, names       = NULL
, dimnames    = NULL
, ramclass    = NULL
, ramattribs  = NULL
, vmode       = NULL
, update      = NULL
, pattern     = NULL
, filename    = NULL
, overwrite   = FALSE
, readonly    = FALSE
, pagesize    = NULL  # getOption("ffpagesize")
, caching     = NULL  # getOption("ffcaching")
, finalizer   = NULL
, finonexit   = NULL  # getOption("fffinonexit")
, FF_RETURN   = TRUE
, BATCHSIZE   = .Machine$integer.max
, BATCHBYTES  = getOption("ffbatchbytes")
, VERBOSE     = FALSE
)

Arguments

Details

The atomic data is stored in filename as a native encoded raw flat file on disk, OS specific limitations of the file system apply. The number of elements per ff object is limited to the integer indexing, i.e. .Machine$integer.max. Atomic objects created with ff are is.open, a C++ object is ready to access the file via memory-mapping. Currently the C++ backend provides two caching schemes: 'mmnoflush' let the OS decide when to flash memory mapped pages and 'mmeachflush' will flush memory mapped pages at each page swap per ff file. These minimal memory ressources can be released by closeing or deleteing the ff file. ff objects can be saved and loaded across R sessions. If the ff file still exists in the same location, it will be opened automatically at the first attempt to access its data. If the ff object is removed, at the next garbage collection (see gc) the ff object's finalizer is invoked. Raw data files can be made accessible as an ff object by explicitly given the filename and vmode but no size information (length or dim). The ff object will open the file and handle the data with respect to the given vmode. The close finalizer will close the ff file, the delete finalizer will delete the ff file. The default finalizer deleteIfOpen will delete open files and do nothing for closed files. If the default finalizer is used, two actions are needed to protect the ff file against deletion: create the file outside the standard 'fftempdir' and close the ff object before removing it or before quitting R. When R is exited through q, the finalizer will be invoked depending on the 'fffinonexit' option, furthermore the 'fftempdir' is unlinked.

Value

If (!FF_RETURN) then a ram object like those generated by vector, matrix, array but with attributes 'vmode', 'physical' and 'virtual' accessible via vmode, physical and virtual
If (FF_RETURN) an object of class 'ff' which is a a list with two components:

Physical object component

The 'ff_pointer' carries the following 'physical' or readonly attributes, which are accessible via physical:

Virtual object component

The 'virtual' component carries the following attributes (some of which might be NULL):

Class

You should not rely on the internal structure of ff objects or their ram versions. Instead use the accessor functions like vmode, physical and virtual. Still it would be wise to avoid attributes AND classes 'vmode', 'physical' and 'virtual' in any other packages. Note that the 'ff' object's class attribute also has copy-by-value semantics ('virtual'). For the 'ff' object the following class attritibutes are known:

Methods

The following methods and functions are available for ff objects:

ff options

Through options or getOption one can change and query global features of the ff package:

OS specific

The following table gives an overview of file size limits for common file systems (see https://en.wikipedia.org/wiki/Comparison_of_file_systems for further details):

Credits

Package Version 1.0

Package Version 2.0

Licence

Package under GPL-2, included C++ code released by Daniel Adler under the less restrictive ISCL

Note

Note that the standard finalizers are generic functions, their dispatch to the 'ff_pointer' method happens at finalization time, their 'ff' methods exist for direct calling.

See Also

vector, matrix, array, as.ff, as.ram

Examples

message("make sure you understand the following ff options 
    before you start using the ff package!!")
  oldoptions <- options(fffinalizer="deleteIfOpen", fffinonexit="TRUE", fftempdir=tempdir())
  message("an integer vector")
  ff(1:12)                  
  message("a double vector of length 12")
  ff(0, 12)
  message("a 2-bit logical vector of length 12 (vmode='boolean' has 1 bit)")
  ff(vmode="logical", length=12)
  message("an integer matrix 3x4 (standard colwise physical layout)")
  ff(1:12, dim=c(3,4))
  message("an integer matrix 3x4 (rowwise physical layout, but filled in standard colwise order)")
  ff(1:12, dim=c(3,4), dimorder=c(2,1))
  message("an integer matrix 3x4 (standard colwise physical layout, but filled in rowwise order
aka matrix(, byrow=TRUE))")
  ff(1:12, dim=c(3,4), bydim=c(2,1))
  gc()
  options(oldoptions)

  if (ffxtensions()){
     message("a 26-dimensional boolean array using 1-bit representation
      (file size 8 MB compared to 256 MB int in ram)")
     a <- ff(vmode="boolean", dim=rep(2, 26))
     dimnames(a) <- dummy.dimnames(a)
     rm(a); gc()
  }

  ## Not run: 

     message("This 2GB biglm example can take long, you might want to change
       the size in order to define a size appropriate for your computer")
     require(biglm)

     b <- 1000
     n <- 100000
     k <- 3
     memory.size(max = TRUE)
     system.time(
     x <- ff(vmode="double", dim=c(b*n,k), dimnames=list(NULL, LETTERS[1:k]))
     )
     memory.size(max = TRUE)
     system.time(
     ffrowapply({
        l <- i2 - i1 + 1
        z <- rnorm(l)
        x[i1:i2,] <- z + matrix(rnorm(l*k), l, k)
     }, X=x, VERBOSE=TRUE, BATCHSIZE=n)
     )
     memory.size(max = TRUE)

     form <- A ~ B + C
     first <- TRUE
     system.time(
     ffrowapply({
        if (first){
          first <- FALSE
          fit <- biglm(form, as.data.frame(x[i1:i2,,drop=FALSE], stringsAsFactors = TRUE))
        }else
          fit <- update(fit, as.data.frame(x[i1:i2,,drop=FALSE], stringsAsFactors = TRUE))
     }, X=x, VERBOSE=TRUE, BATCHSIZE=n)
     )
     memory.size(max = TRUE)
     first
     fit
     summary(fit)
     rm(x); gc()
  
## End(Not run)

Description

The ffapply functions support convenient batched processing of ff objects such that each single batch or chunk will not exhaust RAM and such that batchs have sizes as similar as possible, see bbatch. Differing from R's standard apply which applies a FUNction, the ffapply functions do apply an EXPRession and provide two indices FROM="i1" and TO="i2", which mark beginning and end of the batch and can be used in the applied expression. Note that the ffapply functions change the two indices in their parent frame, to avoid conflicts you can use different names through FROM="i1" and TO="i2". For support of creating return values see details.

Usage

ffvecapply(EXPR, X = NULL, N = NULL, VMODE = NULL, VBYTES = NULL, RETURN = FALSE
, CFUN = NULL, USE.NAMES = TRUE, FF_RETURN = TRUE, BREAK = ".break"
, FROM = "i1", TO = "i2"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)
ffrowapply(EXPR, X = NULL, N = NULL, NCOL = NULL, VMODE = NULL, VBYTES = NULL
, RETURN = FALSE, RETCOL = NCOL, CFUN = NULL, USE.NAMES = TRUE, FF_RETURN = TRUE
, FROM = "i1", TO = "i2"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)
ffcolapply(EXPR, X = NULL, N = NULL, NROW = NULL, VMODE = NULL, VBYTES = NULL
, RETURN = FALSE, RETROW = NROW, CFUN = NULL, USE.NAMES = TRUE, FF_RETURN = TRUE
, FROM = "i1", TO = "i2"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)
ffapply(EXPR = NULL, AFUN = NULL, MARGIN = NULL, X = NULL, N = NULL, DIM = NULL
, VMODE = NULL, VBYTES = NULL, RETURN = FALSE, CFUN = NULL, USE.NAMES = TRUE
, FF_RETURN = TRUE, IDIM = "idim"
, FROM = "i1", TO = "i2", BREAK = ".break"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)

Arguments

Details

ffvecapply is the simplest ffapply method for ff_vectors. ffrowapply and ffcolapply is for ff_matrix, and ffapply is the most general method for ff_arrays and ff_vectors.
There are many ways to change the return value of the ffapply functions. In its simplest usage – batched looping over an expression – they don't return anything, see invisible. If you switch RETURN=TRUE in ffvecapply then it is assumed that all looped expressions together return one vector of length N, and via parameter FF_RETURN, you can decide whether this vector is in ram or is an ff object (or even which ff object to use). ffrowapply and ffcolapply additionally have parameter RETCOL resp. RETROW which defaults to returning a matrix of the original size; in order to just return a vector of length N set this to NULL, or specify a number of columns/rows for the return matrix. It is assumed that the expression will return appropriate pieces for this return structure (see examples). If you specify RETURN=TRUE and a collapsing function name CFUN, then it is assumed that the batched expressions return aggregated information, which is first collected in a list, and finally the collapsing function is called on this list: do.call(CFUN, list). If you want to return the unmodified list, you have to specify CFUN="list" for obvious reasons.
ffapply allows usages not completly unlike apply: you can specify the name of a function AFUN to be applied over MARGIN. However note that you must specify RETURN=TRUE in order to get a return value. Also note that currently ffapply assumes that your expression returns exactly one value per cell in DIM[MARGINS]. If you want to return something more complicated, you MUST specify a CFUN="list" and your return value will be a list with dim attribute DIM[MARGINS]. This means that for a function AFUN returning a scalar, ffapply behaves very similar to apply, see examples. Note also that ffapply might create a object named '.ffapply.dimexhausted' in its parent frame, and it uses a variable in the parent frame for loop-switching between dimensions, the default name 'idim' can be changed using the IDIM parameter. Finally you can break out of the implied loops by assigning TRUE to a variable with the name in BREAK.

Value

see details

Note

xx The complete generation of the return value is preliminary and the arguments related to defining the return value might still change, especially ffapply is work in progress

Author(s)

Jens Oehlschlägel

See Also

apply, expression, bbatch, repfromto, ffsuitable

Examples

message("ffvecapply examples")
   x <- ff(vmode="integer", length=100)
   message("loop evaluate expression without returning anything")
   ffvecapply(x[i1:i2] <- i1:i2, X=x, VERBOSE=TRUE)
   ffvecapply(x[i1:i2] <- i1:i2, X=x, BATCHSIZE=20, VERBOSE=TRUE)
   ffvecapply(x[i1:i2] <- i1:i2, X=x, BATCHSIZE=19, VERBOSE=TRUE)
   message("lets return the combined expressions as a new ff object")
   ffvecapply(i1:i2, N=length(x), VMODE="integer", RETURN=TRUE, BATCHSIZE=20)
   message("lets return the combined expressions as a new ram object")
   ffvecapply(i1:i2, N=length(x), VMODE="integer", RETURN=TRUE, FF_RETURN=FALSE, BATCHSIZE=20)
   message("lets return the combined expressions in existing ff object x")
   x[] <- 0L
   ffvecapply(i1:i2, N=length(x), VMODE="integer", RETURN=TRUE, FF_RETURN=x, BATCHSIZE=20)
   x
   message("aggregate and collapse")
   ffvecapply(summary(x[i1:i2]), X=x, RETURN=TRUE, CFUN="list", BATCHSIZE=20)
   ffvecapply(summary(x[i1:i2]), X=x, RETURN=TRUE, CFUN="crbind", BATCHSIZE=20)
   ffvecapply(summary(x[i1:i2]), X=x, RETURN=TRUE, CFUN="cmean", BATCHSIZE=20)

   message("how to do colSums with ffrowapply")
   x <- ff(1:1000, vmode="integer", dim=c(100, 10))
   ffrowapply(colSums(x[i1:i2,,drop=FALSE]), X=x, RETURN=TRUE, CFUN="list", BATCHSIZE=20)
   ffrowapply(colSums(x[i1:i2,,drop=FALSE]), X=x, RETURN=TRUE, CFUN="crbind", BATCHSIZE=20)
   ffrowapply(colSums(x[i1:i2,,drop=FALSE]), X=x, RETURN=TRUE, CFUN="csum", BATCHSIZE=20)

   message("further ffrowapply examples")
   x <- ff(1:1000, vmode="integer", dim=c(100, 10))
   message("loop evaluate expression without returning anything")
   ffrowapply(x[i1:i2, ] <- i1:i2, X=x, BATCHSIZE=20)
   message("lets return the combined expressions as a new ff object (x unchanged)")
   ffrowapply(2*x[i1:i2, ], X=x, RETURN=TRUE, BATCHSIZE=20)
   message("lets return a single row aggregate")
   ffrowapply(t(apply(x[i1:i2,,drop=FALSE], 1, mean)), X=x, RETURN=TRUE, RETCOL=NULL, BATCHSIZE=20)
   message("lets return a 6 column aggregates")
   y <- ffrowapply( t(apply(x[i1:i2,,drop=FALSE], 1, summary)), X=x
   , RETURN=TRUE, RETCOL=length(summary(0)), BATCHSIZE=20)
   colnames(y) <- names(summary(0))
   y
   message("determine column minima if a complete column does not fit into RAM")
   ffrowapply(apply(x[i1:i2,], 2, min), X=x, RETURN=TRUE, CFUN="pmin", BATCHSIZE=20)

   message("ffapply examples")
   x <- ff(1:720, dim=c(8,9,10))
   dimnames(x) <- dummy.dimnames(x)
   message("apply function with scalar return value")
   apply(X=x[], MARGIN=3:2, FUN=sum)
   apply(X=x[], MARGIN=2:3, FUN=sum)
   ffapply(X=x, MARGIN=3:2, AFUN="sum", RETURN=TRUE, BATCHSIZE=8)
   message("this is what CFUN is based on")
   ffapply(X=x, MARGIN=2:3, AFUN="sum", RETURN=TRUE, CFUN="list", BATCHSIZE=8)

   message("apply functions with vector or array return value currently have limited support")
   apply(X=x[], MARGIN=3:2, FUN=summary)
   message("you must use CFUN, the rest is up to you")
   y <- ffapply(X=x, MARGIN=3:2, AFUN="summary", RETURN=TRUE, CFUN="list", BATCHSIZE=8)
   y
   y[[1]]

   rm(x); gc()