Save very large matrix

Hi, I am trying to save a large matrix (40,000 x 50,000) into a matrix file (.fmt) but I get the error G0132: Can't write, disk probably full. The disk is not full. Is there some limit to the size of the matrices or the matrix files?

 

1 Answer



0



That matrix is about 16 GB. That is too large for the .fmt file format. You can save it as either an HDF5 file, or a GAUSS Data Archive.

Example of creating an HDF5 file

x = { 1 2 3,
      4 5 6,
      7 8 9 };

fname = "mydata.h5";

//HDF5 files can hold many different datasets/matrics
dataset = "/data";

//Create HDF5 file to hold data
h5create(fname, dataset, rows(x)|cols(x));

//Add data to file
h5write(fname, dataset, x);

//Optional step, add variable names
var_names = "height" $| "weight" $| "age";
h5writeAttribute(fname, dataset, "headers", var_names);

//Then you can load the data, using 'loadd'
//Example loadding all columns
x_new = loadd("h5://mydata.h5/data");

//Example loading columns by name
ht_wt = loadd("h5://mydata.h5/data", "height + weight");

Example using GAUSS Data Archive

x = { 1 2 3,
      4 5 6,
      7 8 9 };

fname = "mydata.gda";

//Create GDA file
ret = gdaCreate(fname, 1);

//Add data to GDA file
matrix_name = "x";
ret = gdaWrite(fname, x, matrix_name); 

//Read 'x' from GDA file
x_new = gdaRead(fname, matrix_name);

aptech

1,773

Your Answer

1 Answer

0

That matrix is about 16 GB. That is too large for the .fmt file format. You can save it as either an HDF5 file, or a GAUSS Data Archive.

Example of creating an HDF5 file

x = { 1 2 3,
      4 5 6,
      7 8 9 };

fname = "mydata.h5";

//HDF5 files can hold many different datasets/matrics
dataset = "/data";

//Create HDF5 file to hold data
h5create(fname, dataset, rows(x)|cols(x));

//Add data to file
h5write(fname, dataset, x);

//Optional step, add variable names
var_names = "height" $| "weight" $| "age";
h5writeAttribute(fname, dataset, "headers", var_names);

//Then you can load the data, using 'loadd'
//Example loadding all columns
x_new = loadd("h5://mydata.h5/data");

//Example loading columns by name
ht_wt = loadd("h5://mydata.h5/data", "height + weight");

Example using GAUSS Data Archive

x = { 1 2 3,
      4 5 6,
      7 8 9 };

fname = "mydata.gda";

//Create GDA file
ret = gdaCreate(fname, 1);

//Add data to GDA file
matrix_name = "x";
ret = gdaWrite(fname, x, matrix_name); 

//Read 'x' from GDA file
x_new = gdaRead(fname, matrix_name);


You must login to post answers.

Have a Specific Question?

Get a real answer from a real person

Need Support?

Get help from our friendly experts.