有效地将带时间戳的传感器数据组合到R中的事件中

Question

底层数据集由传感器生成.每6秒钟,每个传感器发送一个信号,识别范围内的所有人(有FOB).无视人员,典型数据如下所示:

   SensorID   timestamp
     2        2015-08-04 09:56:32
     2        2015-08-04 09:56:38
     2        2015-08-05 18:45:20
     3        2015-08-04 09:54:33
     3        2015-08-04 09:54:39
     3        2015-08-04 09:57:31
     3        2015-08-04 09:58:09
     3        2015-08-04 09:58:15
     3        2015-08-04 09:58:33
     3        2015-08-04 09:58:39

我想将此转换为具有开始和结束时间的事件,其中来自相同传感器(和fob)的连续信号被认为是相同事件的一部分,如果它们相隔小于60秒.

因此上述测试数据将转换为:

SensorID  startTime              endTime              sensorCount   duration
2         2015-08-04 09:56:32    2015-08-04 09:56:38  2             6 secs
2         2015-08-05 18:45:20    2015-08-05 18:45:20  1             0 secs
3         2015-08-04 09:54:33    2015-08-04 09:54:39  2             6 secs
3         2015-08-04 09:57:31    2015-08-04 09:58:39  5             68 secs

我有适用的代码.

# identify the ends of sequences
lastKeep <- df$SensorID != df$SensorID[-1L] |
  difftime(df$timestamp[-1L], df$timestamp, units = "secs") > 60

# set startTime and cumulative time and number of signals
df$startTime <- df$timestamp
df$endTime <- df$timestamp
df$sensorCount <- 1

for(jj in 2:nrow(df)) {
 if (lastKeep[jj-1] == FALSE) {
   df$startTime[jj] = df$startTime[jj-1]
   df$sensorCount[jj] = df$sensorCount[jj-1] + 1
 }
}

# select combined records and create duration
df <- df[lastKeep,]
df$duration <- difftime(df$endTime, df$startTime, units = "secs")
df$timestamp <- NULL

但是,对于2000条记录的实际测试数据,此代码需要几秒钟,而完整数据集已经有650万条记录,仍在收集中.因此,我需要一些有效的东西.

有没有办法对此进行矢量化,尽管它依赖于"先前"记录来提供累积时间和信号数量？

我目前的计划是使用Rcpp,但我的C++技能充其量只是平庸.或者,是否存在可以折叠连续信号记录的R包？我在时间序列或信号处理领域找不到它,但它们不是我的领域所以我可能错过了一些明显的东西.

Answer 1

这是data.table使用GH上的devel版本的可能解决方案,它应该足够有效

library(data.table) #V 1.9.5+
setDT(df)[, timestamp := as.POSIXct(timestamp)] # Make sure it's a valid POSIXct class
df[, .(
       startTime = timestamp[1L], 
       endTime = timestamp[.N], 
       sensorCount = .N,  
       duration = difftime(timestamp[.N], timestamp[1L], units = "secs")
      ), 
      by = .(SensorID, 
       cumsum(difftime(timestamp, shift(timestamp, fill = timestamp[1L]), "secs") > 60))]
#    SensorID cumsum           startTime             endTime sensorCount duration
# 1:        2      0 2015-08-04 09:56:32 2015-08-04 09:56:38           2   6 secs
# 2:        2      1 2015-08-05 18:45:20 2015-08-05 18:45:20           1   0 secs
# 3:        3      1 2015-08-04 09:54:33 2015-08-04 09:54:39           2   6 secs
# 4:        3      2 2015-08-04 09:57:31 2015-08-04 09:58:39           5  68 secs

这里的想法是在每个传感器内按60秒的累积时间差分组,然后分配第一个和最后一个时间戳,组计数,每组的第一个和最后一个时间戳之间的差异.

Answer 2

...和dplyr(+ lubridate)方法,假设dt是上面提供的数据集:

library(dplyr)
library(lubridate)


dt %>% 
  mutate(timestamp = ymd_hms(timestamp)) %>%
  group_by(SensorID) %>%                                                       # for each sensor
  mutate(dist = as.numeric(difftime(timestamp,                                 # create distance between consecutive signals 
                                    lag(timestamp, default=min(timestamp)),           
                                    units = "secs"))) %>%
  mutate(flag = ifelse(dist > 60, 1, 0),                                       # flag distances > 60''
         sessionID = cumsum(flag)+1) %>%                                       # create session id
  group_by(SensorID, sessionID) %>%                                            # for each sensor and session
  summarise(startTime = min(timestamp),                                        # get start, end and counts
            endTime = max(timestamp),
            sensorCount = n()) %>%
  mutate(duration = difftime(endTime, startTime, units="secs")) %>%            # get duration
  ungroup()

#   SensorID sessionID           startTime             endTime sensorCount duration
# 1        2         1 2015-08-04 09:56:32 2015-08-04 09:56:38           2   6 secs
# 2        2         2 2015-08-05 18:45:20 2015-08-05 18:45:20           1   0 secs
# 3        3         1 2015-08-04 09:54:33 2015-08-04 09:54:39           2   6 secs
# 4        3         2 2015-08-04 09:57:31 2015-08-04 09:58:39           5  68 secs