Re: Problem with storage of large data

[Aurel Page <aurel.page@normalesup.org>]

    Dear E.D.,
    
    You should
    1/ use readvec for large data
    2/ not perform operations in data files (in particular, do not store
    polynomials as sums of products coef * x^i)
    
    Example:
    
    \\fake data
    nbpol = 700;
    dgpol = 71;
    lgpol = dgpol+1;
    ncoefs = nbpol*lgpol;
    vN = vector(ncoefs,i,random(10^10000));
    vD = vector(ncoefs,i,random(10^10000));
    for(i=1,ncoefs,write("fileN",vN[i]));
    for(i=1,ncoefs,write("fileD",vD[i]));
    
    \\read data
    vN = readvec("fileN");
    vD = readvec("fileD");
    vrat = vector(ncoefs,i,vN[i]/vD[i]);
    vpol = vector(nbpol,i,Pol(vrat[lgpol*(i-1)+1 .. lgpol*i]));
    
    \\basic sanity check
    Set(apply(poldegree,vpol))
    
    Maybe a file in readvec format with each line being Pol([ list of
    coefficients as rational numbers ]) would also be ok, I have not
    tried.
    
    This could also be made more efficient in library mode (avoid
    copies, do not compute gcd between numerator and denominator, etc).
    
    Cheers,
    Aurel

On 25/08/2025 11:12, Ewan Delanoy wrote:

I've got a large (1.2 GB, around 100000 lines) gp file that the GP interpreter cannot read - when I call `read` on that file the interpreter, after a few minutes, prints "Killed" and exits suddenly.

The issue is only one of file size, because the operations in the file are very elementary - it only uses addition, multiplication and accessing/setting elements of an array.

My intent was to store the results of somewhat long computations. The values involved are all univariate polynomials, and the heavy core of the data is a set of about 700 polynomials of degree 71, whose coefficients are rationals whose numerators and denominators have up to 10000 digits.

When I simply and naively use the `write` function to put all this data into a file, I get this

large GP-unreadable file.

Any ideas on how to make this storage work ?

E. D.