Saving each step

A common way to run HydroBOT is to use a single document (i.e. not running the Controller, Aggregator, and Comparer as separate notebooks), but save the outputs at each step. This allows re-starting analyses if necessary and is particularly useful for large jobs, especially if batching over many scenarios. That said, in practice, once batching takes more than several minutes per stage, it often does make most sense to split into separate notebooks or scripts for each step.

Retaining everything in-memory is a very simple flip of a switch, demoed in its own doc, which parallels this one nearly exactly with the exception of not specifying an outputType in prep_run_save_ewrs() or a savepath in read_and_agg(). Saving is generally how workflows are run for large projects, and so is the example used here for getting started.

Load the package

Directories

Here, we will use the example hydrographs that come with HydroBOT.

Normally project_dir and hydro_dir would point somewhere external (and typically, having hydro_dir inside project_dir will make life easier).

hydro_dir <- system.file("extdata/testsmall/hydrographs", package = "HydroBOT")

project_dir <- file.path("test_dir")

# Generated data
# EWR outputs (will be created here in controller, read from here in aggregator)
ewr_results <- file.path(project_dir, "module_output", "EWR")

# outputs of aggregator. There may be multiple modules
agg_results <- file.path(project_dir, "aggregator_output")

We need the ‘yearly’ EWR outputs for HydroBOT processing, though we can get the EWR to return any of its options.

Note

If this is your first time using HydroBOT, and you haven’t set up a Python environment, this might take a while as {reticulate} does it for you. This is required because the EWR tool is in Python.

ewr_out <- prep_run_save_ewrs(
  hydro_dir = hydro_dir,
  output_parent_dir = project_dir,
  outputType = list("yearly"),
  returnType = list("yearly")
)

Aggregation

We need to define an aggregation sequence, which specifies the steps along space, time, and theme dimensions, as well as a matching sequence of aggregation functions at each step. The example here is a reasonable default for many situations, but should be considered carefully. Please see the aggregator section for a detailed treatment of the syntax and capabilities of these sequences.

For this example, if any of the ‘versions’ of an EWR pass, say it passes in step 2. If we think the ‘versions’ are actually separate EWRs for different purposes, we might use ‘ArithmeticMean’ in step 2 intead.

aggseq <- list(
  all_time = "all_time",
  ewr_code = c("ewr_code_timing", "ewr_code"),
  env_obj = c("ewr_code", "env_obj"),
  sdl_units = sdl_units,
  Target = c("env_obj", "Target"),
  mdb = basin,
  target_5_year_2024 = c("Target", "target_5_year_2024")
)

funseq <- list(
  all_time = "ArithmeticMean",
  ewr_code = "CompensatingFactor",
  env_obj = "ArithmeticMean",
  sdl_units = "ArithmeticMean",
  Target = "ArithmeticMean",
  mdb = "SpatialWeightedMean",
  target_5_year_2024 = "ArithmeticMean"
)

Run the aggregation. Use the auto_ewr_PU shortcut to auto-specify some needed grouping for EWR outputs (see here for more detail).

Read the messages. Most of these are saying it’s more explicit to avoid auto_ewr_PU = TRUE, but there are also messages saying that the causal network is missing links.

aggout <- read_and_agg(
  datpath = ewr_results,
  type = "achievement",
  geopath = bom_basin_gauges,
  causalpath = causal_ewr,
  groupers = "scenario",
  aggCols = "ewr_achieved",
  auto_ewr_PU = TRUE,
  aggsequence = aggseq,
  funsequence = funseq,
  saveintermediate = TRUE,
  namehistory = FALSE,
  keepAllPolys = FALSE,
  returnList = TRUE,
  add_max = FALSE,
  savepath = agg_results
)
ℹ EWR outputs auto-grouped
• Done automatically because `auto_ewr_PU = TRUE`
• EWRs should be grouped by `SWSDLName`, `planning_unit_name`, and `gauge` until aggregated to larger spatial areas.
• Rows will collapse otherwise, silently aggregating over the wrong dimension
• Best to explicitly use `group_until` in `multi_aggregate()` or `read_and_agg()`.
ℹ EWR outputs auto-grouped
• Done automatically because `auto_ewr_PU = TRUE`
• EWRs should be grouped by `SWSDLName`, `planning_unit_name`, and `gauge` until aggregated to larger spatial areas.
• Rows will collapse otherwise, silently aggregating over the wrong dimension
• Best to explicitly use `group_until` in `multi_aggregate()` or `read_and_agg()`
.
ℹ EWR outputs auto-grouped
• Done automatically because `auto_ewr_PU = TRUE`
• EWRs should be grouped by `SWSDLName`, `planning_unit_name`, and `gauge` until aggregated to larger spatial areas.
• Rows will collapse otherwise, silently aggregating over the wrong dimension
• Best to explicitly use `group_until` in `multi_aggregate()` or `read_and_agg()`
.
ℹ EWR gauges joined to larger units pseudo-spatially.
• Done automatically because `auto_ewr_PU = TRUE`
• Non-spatial join needed because gauges may inform areas they are not within
• Best to explicitly use `pseudo_spatial = 'sdl_units'` in `multi_aggregate()` or `read_and_agg()`.

! Unmatched links in causal network
• 4 from Target to target_5_year_2024

Comparer

Now we can make a couple quick plots to see what we’ve made. For more detail about plotting options and controls, see the comparer.

Maps and spatial scaling

map_example <- aggout$sdl_units |>
  dplyr::filter(env_obj == "NF1") |> # Need to reduce dimensionality
  plot_outcomes(
    outcome_col = "ewr_achieved",
    plot_type = "map",
    colorset = "ewr_achieved",
    pal_list = list("scico::lapaz"),
    pal_direction = -1,
    facet_col = "scenario",
    facet_row = "env_obj",
    sceneorder = c("down4", "base", "up4"),
    underlay_list = "basin"
  ) +
  ggplot2::theme(legend.position = "bottom")

map_example

Bars- SDL units and scenarios

SDL unit differences across all environmental objectives

catchcompare <- aggout$env_obj |>
  plot_outcomes(
    outcome_col = "ewr_achieved",
    colorset = "SWSDLName",
    pal_list = list("calecopal::lake"),
    sceneorder = c("down4", "base", "up4"),
    position = "dodge"
  )

catchcompare