Saving Results

GridSearchSys objects can be saved to file with the Serialization library and a bit of overhead work, encapsulated in save_serde_data and load_serde_data.

PowerSystemsExperiments.save_serde_data — Function

save_serde_data(gss::GridSearchSys, path::String)

serializes the GridSearchSys and saves it to path to be read later using Serialization.deserialize (through load_serde_data)

deletes all .jls files and the .gss file in path if path is a directory containing those files.

source

PowerSystemsExperiments.load_serde_data — Function

load_serde_data(path::String; load_gss::Bool=true)

Loads serialized data from file or folder.

If path is a folder (indented use):

read all non-hidden .jls files and concatenate the DataFrames from them
if .hfile exists, read it to define any necessary functions (not fully working)
if load_gss is true and .gss exists, read it to load the GridSearchSys object and attach the DataFrame to it.

then, return either the concatenated DataFrame or the GridSearchSys object containing it. If path is a file (backup data recovery option):

deserialize file
return the result

source

They both save to or load from a directory, since they work with multiple files.

Saving to file lets you do neat things like

gss = load_serde_data("path/to/gss/folder")
add_result!(gss, "eigenvalues", eu.get_eigs)
save_serde_data("path/to/gss/folder")

In addition, the set_chunksize! method controls chunking.

PowerSystemsExperiments.set_chunksize! — Function

set_chunksize!(gss::GridSearchSys, chunksize::Union{Int, Float64})

set the number of rows save in each file (and thus how many to hold in memory before saving to file).

Set to Inf to hold all rows in memory (useful for small datasets and to allow use of the dataframe immediately after running the sims)

source

The default chunk size is Inf. If this is the case, then when save_serde_data is called, it will just save the entire dataframe of results to a single file. Otherwise, it will chunk it into files with gss.chunksize rows.

When execute_sims! is called, it writes the results to gss.df one row at a time. When it reaches gss.chunksize total rows, it saves the entire dataframe to file, then deletes it from RAM. This allows very very large sweeps to be run in finite memory.

When it finishes all of the simulations, it saves whatever is left to file as well.

To prevent it from saving anything, you can run

set_chunksize(gss, Inf)

The header file and associated hacks

One issue with serialization is that it doesn't work with user-defined functions. For example, if we tried to save our gss from before, it would save some reference to the set_power_setpt! function, but wouldn't save it. Therefore, we need to redefine it wherever we want to load back our results.

But unless you're trying to run the simulations again, you don't really need the function definition. You just need something for it to refer to. As a result, my quick hack is that every time we reference an external function, we add "function {name} end; " to the gss.hfile variable. Then we eval this string when we load the gss object back, and it all works!

...In theory, at least. The best solution is always to simply import or copy the definitions into wherever you need them. If you are having issues, you can typically just open the file (.hfile in the save directory) and delete whichever functions are screaming at you.

If all else fails

Because this functionality is still in development, it has been designed to allow you to bypass everything and get to your data no matter what happens. If the data fails to load, you can just load the dataframe with the results without the GridSearchSys object. This means you won't be able to use add_results!, but you can still do the same thing manually, since all the data will be present.

The load_serde_data function will also accept a path to one of the .jls files saved by save_serde_data. These are just plain serialized DataFrames, and they load back quite reliably.

If you don't have to do this, avoid it, but it's there just in case.