• mario.six@gdsys.cc's avatar
    rsa: Fix return value and masked error · 2b9ec762
    mario.six@gdsys.cc authored
    When signing images, we repeatedly call fit_add_file_data() with
    successively increasing size values to include the keys in the DTB.
    
    Unfortunately, if large keys are used (such as 4096 bit RSA keys), this
    process fails sometimes, and mkimage needs to be called repeatedly to
    integrate the keys into the DTB.
    
    This is because fit_add_file_data actually returns the wrong error
    code, and the loop terminates prematurely, instead of trying again with
    a larger size value.
    
    This patch corrects the return value by fixing the return value of
    fdt_add_bignum, fixes a case where an error is masked by a unconditional
    setting of a return value variable, and also removes a error message,
    which is misleading, since we actually allow the function to fail. A
    (hopefully helpful) comment is also added to explain the lack of error
    message.
    
    This is probably related to 1152a05e ("tools: Correct error handling in
    fit_image_process_hash()") and the corresponding error reported here:
    
    https://www.mail-archive.com/u-boot@lists.denx.de/msg217417.htmlSigned-off-by: default avatarMario Six <mario.six@gdsys.cc>
    2b9ec762
rsa-sign.c 10.7 KB