There has been entries of past editions of NaNoRenO that were made in RPG Maker, likeSymbiosis.

Also, inthis past thread, sakevisual explicitly says it is fine, with the following caveat:

RPG Maker is definitely allowed for this! Definitions of a visual novel are, as always, very subjective, but for a rough rule of thumb, 70% or more of gameplay should be dialogue or text-based narration with the traditional VN format of text box + full screen sprites.