Load data within a loop

Hi,

I was wondering if its possible to load a different matrix in each iteration of a loop. I can't load the full data together as GAUSS runs into "error G0030 : Insufficient memory" after a few iterations. As I will need matrix1 only in iteration1 and so on, I wanted to reuse the space allotted to the matrix in each iteration by loading a different matrix, but storing it under the same name (all the matrices have the same dimension). However, I am not sure if there is a way to dynamically change the load statement so that it reads different files in each iteration.

Any suggestion on  how to handle this issue would be greatly appreciated.

Thanks

4 Answers



0



Do all these matrices have the same number of columns and same column types?

aptech

1,773


0



Yes, they do.



0



All the matrices have the same number of rows and columns and same column types.

I would greatly appreciate any suggestion in this issue.

Thanks



0



If they all have the same number of columns, you can the readr function to load sequential chunks of rows from a dataset. For example:

new;

/*
** Create dataset for example
*/
rndseed 432143;
X = seqa(1, 1, 10) ~ rndn(10,3);

fname = "test_data.dat";
call saved(X, fname, 0);

// Open file handle to access dataset
fh = dataOpen(fname, "read");

// Load 2 rows per iteration
for i(1, 5, 1);
    X_tmp = readr(fh, 2);

    print "Chunk "$+ntos(i)$+" of "$+ntos(5);
    print X_tmp;
endfor;

call close(fh);

If all the data won't fit in memory at once, you can use dataCreate and writer to create the dataset.

aptech

1,773

Your Answer

4 Answers

0

Do all these matrices have the same number of columns and same column types?

0

Yes, they do.

0

All the matrices have the same number of rows and columns and same column types.

I would greatly appreciate any suggestion in this issue.

Thanks

0

If they all have the same number of columns, you can the readr function to load sequential chunks of rows from a dataset. For example:

new;

/*
** Create dataset for example
*/
rndseed 432143;
X = seqa(1, 1, 10) ~ rndn(10,3);

fname = "test_data.dat";
call saved(X, fname, 0);

// Open file handle to access dataset
fh = dataOpen(fname, "read");

// Load 2 rows per iteration
for i(1, 5, 1);
    X_tmp = readr(fh, 2);

    print "Chunk "$+ntos(i)$+" of "$+ntos(5);
    print X_tmp;
endfor;

call close(fh);

If all the data won't fit in memory at once, you can use dataCreate and writer to create the dataset.


You must login to post answers.

Have a Specific Question?

Get a real answer from a real person

Need Support?

Get help from our friendly experts.