fbmem: don't allow too huge resolutions
authorTetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Wed, 8 Sep 2021 10:27:49 +0000 (19:27 +0900)
committerDaniel Vetter <daniel.vetter@ffwll.ch>
Wed, 8 Sep 2021 16:52:04 +0000 (18:52 +0200)
commit8c28051cdcbe9dfcec6bd0a4709d67a09df6edae
treebf98d1945cca9a2268bff4715044a3542b5d666f
parent87fd9ef47597b2fcee3264eb5f288410b9f376d5
fbmem: don't allow too huge resolutions

syzbot is reporting page fault at vga16fb_fillrect() [1], for
vga16fb_check_var() is failing to detect multiplication overflow.

  if (vxres * vyres > maxmem) {
    vyres = maxmem / vxres;
    if (vyres < yres)
      return -ENOMEM;
  }

Since no module would accept too huge resolutions where multiplication
overflow happens, let's reject in the common path.

Link: https://syzkaller.appspot.com/bug?extid=04168c8063cfdde1db5e
Reported-by: syzbot <syzbot+04168c8063cfdde1db5e@syzkaller.appspotmail.com>
Debugged-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/185175d6-227a-7b55-433d-b070929b262c@i-love.sakura.ne.jp
drivers/video/fbdev/core/fbmem.c