Confirmed using Excel 2010 and NVDA 2017.3 on a Windows 8.1 system. Let me provide a brief summary of the STR employed, results experienced, and other related thoughts.

STR

• Open a blank Excel workbook.
• Press Alt, H, O, P. Note that these keys must be pressed separately respecting the order in which the keys have been specified.
• The Protect Sheet dialog appears and your focus lands on the Password to unprotect sheet edit field.
• Type any password-valid character.
  Expected Result: NVDA reports "star", "circle" or "bullet".
  Actual Result: NVDA announces the entered character itself.
• Press the left arrow to double-check whether or not the character typed previously was converted to an asterisk, bullet, circle, etc..
  Expected and Actual Result: NVDA reports "circle". It is worth noting that this is an inconsistency because NVDA announced the character while typing the password earlier.

In a nutshell, the issue being reported and discussed in this ticket is that NVDA speaks the character being entered in said dialog's said edit control instead of reporting the encrypted shape which the character entered is visually converted to.

Log Snippet

When "a" is the character entered:
IO - inputCore.InputManager.executeGesture (11:14:04.151):
Input: kb(laptop):a
IO - speech._speakSpellingGen (11:14:04.183):
Speaking character u'a'
DEBUG - queueHandler.registerGeneratorObject (11:14:04.183):
Adding generator 12801
DEBUG - queueHandler.pumpAll (11:14:04.197):
generator 12801 finished

Impact

This issue affects all NVDA users also using Excel 2010 and Excel 2013, though this issue may extend to other older and newer versions of Microsoft Excel as well as other Microsoft Office programs. I would like to believe that this bug would be equally applicable to Braille and speech. Excel is a software commonly used in professional and corporate environments, and I would suppose that password protection is a frequently used feature of this program. Verbalization of the password could probably compromise a user's privacy and may be something that could be pretty undesirable in a workplace.
At the same time, a relatively fast speech rate and usage of headphones, which may be typically expected from a professional NVDA user lessen the severity of this bug. In addition, the Synth Settings Ring and Sleep mode could be temporarily used to quickly silence NVDA, type the password, and then bring NVDA back to life, again serving as an inelegant but available work-around.

While this is a security problem, I don't believe that this has ever been expected to work previously. Assigning the feature label, hopefully we will get a chance to address this.

This is still reproducible in Excel 2016 and 365. I think this should be solved soon since especially in organizations we have lots of protected excel sheets.
cc: @michaelDCurran, @lukaszgo1

A user reported "If opening the files by double-clicking on the file, then enter the password NVDA will speak the password. However, open either Word or Excel then open the file NVDA only speaks “star, star etc.”

In trying to reproduce it (I assume they are talking about a password to to open the file - alt+f, i, p then "Encrypt with a password". The original issue here gives the keystrokes to protect a sheet. Additionally you can protect a workbook with alt+r, p, w.

(Protecting a sheet limits the ability to edit cells without the password, protecting the workbook limits adding or deleting worksheets. Confusingly the option on the File - Info pane to require a password on opening the file, and the option on the Review ribbon to stop you deleting worksheets are both called "Protect Workbook")

All of these password options seem to behave the same for me - that is, as the original poster described - characters are read out when typed, but moving back or backspacing results in "circle" being read out. Another thing to note is that visually all three passwords only ever appear as circles. While it is a security concern, it can be mitigated by wearing headphones. (Offering that as a workaround, not a solution)

I'm using NVDA 2020.1 with Office 365 Version: 16.0.12730.20236

I cannot reproduce this in Office 365. Both the protect and unprotect sheet password fields are announced as protected and echo star characters for me. No matter if I open the file from within Excel or from file Explorer.
Either way, Microsoft must expose the protected state if they are not doing so already. I don't think there is an easy way to identify this control otherwise.
The work around would be to temporarily disable typing echo while typing your password into this field.