Playing the piano requires various techniques such as correct keying, fingering and rhythm. Our research group developed a piano learning system to support correct keying and fingering for beginners. However, the system did not support the learning of rhythm. Rhythm consists of various kinds of note and rest, and it is difficult for beginners, who are not used to reading a score, to understand the different duration of each note and rest. Alternatively, there are piano roll scores, which describe timing of keying and releasing clearly, but which do not teach players how to read a musical staff. Therefore, the goal of our study is to construct a piano learning support system that considers rhythm. We discuss methods of effectively indicating information for piano performance, such as rhythm information, while teaching how to read a musical staff. We have developed a prototype system, and evaluated its effectiveness through actual use of the system. We found that it had significant advantages over a piano roll method.

. The average number of keying and rhythm errors
The measurement of rhythm errors and keying errors

Figures - uploaded by Tsutomu Terada

Author content

All figure content in this area was uploaded by Tsutomu Terada

Content may be subject to copyright.

ResearchGate Logo

Discover the world's research

  • 20+ million members
  • 135+ million publications
  • 700k+ research projects

Join for free

_324 _325

[5] K. Pelechrinis, M. Iliofotou, and S. Krishnamurthy,

"Denial of service attacks in wireless networks: The

case of jammers," Communications Surveys Tutorials,

IEEE, vol. 13, no. 2, pp. 245 –257, quarter 2011.

[6] S. Smallwood, D. Trueman, P. R. Cook, and G. Wang,

"Composing for laptop orchestra," Computer Music

Journal, vol. 32, no. 1, pp. 9–25, Mar. 2008.

[7] G. Surges and C. Burns, "Networking infrastructure

for collaborative laptop improvisation," 2008.

[8] G. Wang and P. R. Cook, "ChucK: a concurrent, on-

the-fly, audio programming language," 2003.

[9] M. Wright, "Open sound control: an enabling tech-

nology for musical networking," Organised Sound,

vol. 10, no. 3, pp. 193–200, 2005.

A PIANO LEARNING SUPPORT SYSTEM CONSIDERING RHYTHM

Yoshinari Takegawa

Future University Hakodate

Hakodate, Japan

Tsutomu Terada

Kobe University / PRESTO, JST

Kobe, Japan

Masahiko Tsukamoto

Kobe University

Kobe, Japan

ABSTRACT

Playing the piano requires various techniques such as cor-

rect keying, fingering and rhythm. Our research group

developed a piano learning system to support correct key-

ing and fingering for beginners. However, the system did

not support the learning of rhythm. Rhythm consists of

various kinds of note and rest, and it is difficult for begin-

ners, who are not used to reading a score, to understand

the different duration of each note and rest. Alternatively,

there are piano roll scores, which describe timing of key-

ing and releasing clearly, but which do not teach players

how to read a musical staff. Therefore, the goal of our

study is to construct a piano learning support system that

considers rhythm. We discuss methods of effectively indi-

cating information for piano performance, such as rhythm

information, while teaching how to read a musical staff.

We have developed a prototype system, and evaluated its

effectiveness through actual use of the system. We found

that it had significant advantages over a piano roll method.

1. INTRODUCTION

Piano players need to master various techniques and skills,

such as reading a score, correct keying, proper finger-

ing, correct rhythm (the timing of pressing and releasing

a key), keeping tempo, and dynamics. Players generally

need long-term training. Unfortunately, beginners often

give up because of the difficulty of acquiring these tech-

niques.

Our research group developed a piano learning system

to support correct keying and fingering for beginners[20].

It uses a projector which is set above the keyboard and can

display information along the entire MIDI keyboard. The

proposed system has a fingering check function that uses

the real-time fingering recognition technique that our re-

search group developed [21]. Additionally, we devised

presentation methods to indicate useful information for

piano performances effectively. We place emphasis on

teaching how to read a musical staff in order to enable

learners to be independent from our proposed system af-

ter training.

Another important aspect of performance is rhythm

because it affects performance quality. When learners play

rhythm incorrectly, the performance is awkward even they

press the correct keys. There are various kinds of note and

rest on a score. It is difficult for beginners, who are not

used to reading a score, to understand the different dura-

tion of each note and rest, thus they can learn rhythm most

effectively by using a mechanism that allows them to in-

tuitively understand the different durations. Additionally,

piano performance requires complicated and precise fin-

gering control for each hand in regard to timing. Many

beginners give up playing the piano with both hands due

to the difficulty of the independent movement of each fin-

ger and hand, for example the difference between the tim-

ing of releasing a key with a right-hand finger and that

of a left-hand finger. It is important to make learners un-

derstand their mistakes for example by imposing penal-

ties for errors. The effectiveness of rhythm learning im-

proves through checking mistakes and imposing penalties,

such as the system withholding the next piece of learn-

ing support information when a learner makes a mistake.

Moreover, learners have to acquire proper rhythm as early

as possible since it is difficult for them to rectify their

mistakes once they are accustomed to playing incorrect

rhythm. Furthermore, as the duration of each note and

rest depends on tempo, learners have to be conscious of

this as well.

Our research group developed a piano learning sys-

tem to support correct keying and fingering for begin-

ners. However, the system did not support the learning

of rhythm. Even if users, who are beginners but prac-

tice playing the piano using the proposed system, press

the correct keys with proper fingering in slow tempo with

both hands and can foresee the next keys which are to be

pressed, the performance is awkward because of the in-

correct duration of holding keys and inserting incorrect

rests. This is due to the difficulty of paying attention to

the notes' duration while moving each hand in different

timing. There are piano roll scores, which describe tim-

ing of keying and releasing clearly, but which do not teach

players how to read a musical staff. The musical staff is

the general medium used in musical performance. If be-

ginners cannot read music, they cannot play pieces of mu-

sic which are not stored on the system, without using the

system.

_326 _327

Therefore, the goal of our study is to construct a piano

learning support system that considers rhythm.

We discuss methods to effectively indicate informa-

tion for piano performance, such as rhythm information,

while teaching how to read musical staffs. For example,

the proposed system shows the musical staff with colored

bars layered over the notes and rests to indicate their du-

ration. In this way, learners can understand the duration

of each note and rest intuitively even while playing the pi-

ano. Moreover, the system has a rhythm check function

to allow learners to notice rhythm mistakes and rectify

them, using a metronome function. Learners can flexi-

bly and easily control the speed of the metronome with a

foot pedal.

The remainder of this paper is organized as follows:

Section 2 describes related work, Section 3 explains the

design of the learning support system, Section 4 describes

its implementation, Section 5 explains our evaluation and

discusses the results, and finally Section 6 describes our

conclusions and future work.

2. RELATED WORK

There are many studies of methods to support piano learn-

ers. Piano Tutor[14] is an interactive expert system that

uses with multimedia technology, and has functions such

as automatic page-turning based on score-following tech-

nology, creating performance support information and pre-

senting it with video, music notation, and graphics in re-

sponse to learners' performance. Piano Tutor does not use

a projector to show performance support information, and

the presentation method of Piano Tutor is typically differ-

ence from that of the proposed system. However, Piano

Tutor is a comprehensive learning system, and there is a

possibility that we can develop a more effective learning

system by utilizing Piano Tutor's knowledge.

There are keyboards and software [1, 3, 5] that display

keying position, fingering, and sample videos as support

information during performance. However, these have

problems, such as the lack of a rhythm check function,

as described in Section 1.

PianoTouch[11], ConcertHands[2], and MaGKeyS

Trainer Piano[8] are haptic-based instruction systems for

piano learners. They give a player performance informa-

tion through a tactile feedback unit attached to each finger.

Learners are able to learn keying and fingering techniques

easily but they are forced to wear bulky devices on the

fingers.

Additionally, there are systems that automatically de-

tect the weak points of learners including mis-keying and

fluctuation of tempo or dynamics on the basis of a con-

ventional practice log [12, 16, 17, 19]. There are also

piano lesson support systems [18] that show current ar-

ticulation, agogik, and dynamics. Although these systems

do not have rhythm check functions, we derived useful

knowledge from their development and have put it to use

in our learning support system.

Our research also relates to augmented reality research.

Many new types of projector-based augmented reality sys-

tem [6, 7, 9, 10, 13, 15] have also been proposed. These

works attempt to assist a simple movement-based task.

However, our system supports the learning of an intricate

physical task by tracking the movements associated with

the task and augmenting the physical environment with

prompts and other information to aid the task.

3. SYSTEM DESIGN

As described in Section 1, our research group developed a

piano learning system for beginners to teach correct key-

ing and fingering, as well as how to read a musical staff,

to enable learners to play music, which is not stored in

the system, without the support of the system. However,

the system does not support rhythm. Therefore, we con-

structed a rhythm learning system on the basis of improv-

ing upon the previous system. The proposed system has

presentation methods that help to effectively convey piano

learning, including not only fingering and keying but also

rhythm information (described in Section 3.3 (i)). The

rhythm check function uses a clear presentation method

to allow the learner to recognize and rectify his or her

mistakes (described in Section 3.3 (ii)). Moreover, we

propose a metronome function (described in Section 3.3

(iii)) as well as a function to enhance the usability of the

metronome (described in Section 3.3 (iv) and (v)).

3.1. Previous system

In the previous system, the projector is set above the key-

board and is able to show information along the entire

MIDI keyboard, as shown in Figure 1. Learners find the

piano learning information easy to understand as the pre-

vious system present various kinds of content, such as col-

orful figures and characters in an appropriate position to

allow to learners to see the information easily even while

playing the piano. Additionally, the previous system has

a function that recognizes fingering using a camera[21],

and develops methods for presenting learning support in-

formation for users to check their keying and fingering.

In the following section, we explain the information

presented by the previous system. The letters in Figure 5

correspond to the following list:

(a) NextKey refers to the next key that is to be pressed

in a piece of music, as shown in Figure 1, and each

NextKey is outlined in color to provide keying in-

formation. The NextKeys are indicated by the ar-

rows (a) in Figure 1.

(b) The colors and numbers of the NextKeys are the fin-

gering information. When the NextKey is pressed

using the correct finger, the key is filled in with

the corresponding finger color. The left NextKey is

yellow colored because the correct finger has been

placed on it. On the other hand, when the NextKey

is pressed with the incorrect finger, the key is col-

ored red. When other keys besides the NextKeys

MIDI

Projector

Video Signal PC

Camera

Image

aa

b

c

c

MIDI keyboard

Figure 1. System structure and presented contents of the previous system

Foot pedal

Speed Beat

ⅵ

ⅰ

ⅲ

ⅷ

ⅶ

MIDI

Projector

Video Signal

PC

Foot pedal

information

MIDI keyboard

Figure 2. System structure and presentation of contents of the proposed system

are pressed, these keys are also colored red. In this

way, learners can understand the positions of the

NextKeys, learn fingering technique intuitively, and

rectify their mistakes.

(c) Each musical note is connected to the corresponding

key with a line. This visual support enables learners

to read a score easily, because they can clearly see

the relationship between the musical notes and key

positions

The results of evaluative experiments confirmed that

our system significantly enhanced learning effectiveness

in the early stages of practice, when compared with the

lighted keyboard method which turns the NextKeys red.

3.2. System structure

The structure of the system is shown in Figure 2. The sys-

tem has a foot pedal to control the tempo of the metronome,

and a projector to present learning support information.

The projector is set above the keyboard and can display

information along the entire MIDI keyboard. The sys-

tem uses MIDI data including pitch data and intensity data

from the MIDI keyboard.

3.3. Presented information

We explain the presented information with Figure 2. This

information is updated in sync with the performance. The

Roman numerals in Figure 2correspond to the following

list:

i) The duration bar Rhythm consists of various kinds of

note and rest, and it is important for beginners to un-

derstand the different duration of each one. There-

fore, the proposed system enables learners to under-

stand the duration of each note and rest by showing

colored bars, the lengths of which correspond to the

durations of each note and rest as shown in Figure

3. Additionally, the color of the bar turns from blue

to yellow as the learner holds the key. In this way,

the learner can intuitively understand the remaining

time for which he or she must hold the key. If the

learner holds the key too long the color of the bar

turns from yellow to red and the length of the bar

increases until the learner releases the key.

ii) Rhythm check function The system has a function that

checks the timing of pressing and releasing a key

and whether the key is held for the correct duration.

Moreover, the system checks the timing of press-

ing several keys simultaneously, for example when

_328 _329

[Duration bars before keying]

The color of the bar turns

from blue to yellow as the

learner holds the key.

If the learner holds the key

too long, the color of the bar

turns from yellow to red.

[Holding keys]

Figure 3. Examples of presentation of duration for musi-

cal notations

Cue point

Figure 4. Score with cue points

the learner plays a chord, and the unnecessary rests

between keys or between rests and notes. The er-

ror margin allowed for the timing of actions such as

pressing multiple keys is called the allowable time,

and depends on tempo, the difficulty of the piece of

music and the learning level of the player. The pro-

posed system allows learners to set the allowable

time freely.

iii)The Metronome Users can turn the metronome on or

off. Current tempo and beat are displayed at the dis-

tal ends of the keys shown in Figure 2. The tempo

and the number of beats of the metronome are con-

trolled by pressing the keys that represent current

tempo and beat, respectively.

iv) Control of the metronome using a foot pedal

Different parts of a piece of music have different

degrees of difficulty. When a learner is practicing

difficult parts, he or she tends to play in a slow

tempo at first and then gradually increase the speed.

On the other hand, when the learner practices easy

parts, he or she plays in the tempo indicated by the

score. Therefore, learners can practice a piece of

music more effectively if they have flexible control

of the tempo. We adapted a foot pedal to control

the tempo of the metronome flexibly, and the tempo

gets faster when the learner pedals.

v) Adjustment of the start point of the metronome

There may sometimes be a lot of unexpected pauses

because of the difficulty of playing certain parts of

a piece of music. Additionally, beginners, who are

not used to using one, find it difficult to adjust their

own performance to the sound of a metronome.

Therefore, we propose a function that automatically

adjusts the start point of the metronome to the per-

formance. In this way, beginners do not have to

consider the timing of the metronome, and can start

playing whenever they like.

vi) Presentation of keying position and fingering

This function was also included in the previous sys-

tem. When a key is outlined in color this indicates

that it is the next key that should be pressed. A

number on the key denotes fingering. This function

is useful for beginners, who cannot read out keying

and fingering information from a piece of music

vii) Selection of cue points Users can select cue points

which are indicated on the score by numbers in black

squares as shown in Figure 4. The cue points enable

learners to change the point from which they want

to start practicing. This function is useful when

learners want to practice part of the score again and

again without having to start from the very begin-

ning each time.

viii) Switching of each function These functions are con-

trolled using the keyboard. Keys can be assigned

to commands for operating the system, and an icon

which represents the command assigned to a key is

displayed on the key.

4. IMPLEMENTATION

We implemented a prototype of the piano practice support

system, as described in Section 3.3. We used a SONY

VGNS94PS (Intel Core2 Duo 2.60GHz), running Win-

dows 7, a CASIO PriviA PX-110 equipped with 88 full-

sized keys. We used a BenQ MP776 ST as the projector.

The projected area was 6 octaves (72 keys) and we painted

all the black keys of the MIDI keyboard white. We im-

plemented the system using Microsoft Visual C++.NET

2010 and Intel OpenCV Library. The prototype is shown

in Figure 5.

5. EVALUATION

We conducted an evaluative experiment to investigate the

effectiveness of the proposed system in the beginning stage

of piano performance, when a piano beginner is practicing

the keying, fingering, and rhythm of a new score.

5.1. Experimental Procedure

The evaluation procedure was as follows:

Comparative method In this evaluation, we compared

the proposed method to a piano roll method, and a method

without rhythm support, based on the number of keying

and rhythm errors. Piano roll scores describe timing of

Projector

MIDI keyboard

White board

(as a screen)

PC

Figure 5. Prototype system

keying and releasing clearly, and are used in KEYBOARD

MANIA[4], which enables players who have no formal

musical instrument training to enjoy piano performance

easily. In the piano roll method, each key has a corre-

sponding vertical bar on the screen as shown in Figure 6.

Rectangular icons scroll down the bars to indicate which

keys the learner should press. Users can understand the

duration of each note and rest because the size of the rect-

angular icons is based on the duration of the correspond-

ing notes. Timing is also easy to understand as the user

simply presses the matching keys when the rectangle icons

descend to the bottom of the screen.

Table 1 shows the application of functions for each

method.

In the piano roll method, the system displays not only

a piano roll score but also a musical staff on the piano roll

score. Users are able to see both scores.

The proposed method presented the next learning in-

formation when subjects had pressed a correct key with

correct rhythm, whereas the piano roll method and the

method without rhythm support presented the next infor-

mation when subjects had pressed only a correct key. The

default speed of a metronome is that the duration between

clicks is 0.6sec. One sixteenth note is equal to two clicks.

The Sixteenth note was the smallest note in the trial score.

Subject Nine subjects took part in this experiment, and

there were three subjects for each method. All subjects

Figure 6. An example of a piano roll

belonged to a graduate school of Engineering, had no for-

mal piano training, and were not able to read a score. We

explained how to read pitch and duration from notes and

rests on the musical staff, and how to use the proposed

functions. Moreover, the subjects using the piano roll

method were taught how to read the piano roll as well.

Trial piece The subjects practiced "Piano Sonata No. 11

in A major, K. 331: III (W. A. Mozart)" ,from the begin-

ning to bar 18, as the trial piece for two-handed playing.

The total number of musical notes on the trial piece is 99.

Flow of the evaluation This examination consisted of

two phases: practice and testing. The subjects practiced

the trial piece for 30 minutes during the practice phase.

We instructed them to practice freely. They basically

learned the trial piece by practicing over and over and

using the functions assigned to each method. Then they

played the trial piece from beginning to end in the test

phase, during which the system logged the press and re-

lease timing of the keys. In the test phase, we presented

a score that was the same as the score used in the prac-

tice phase, and this was accompanied by the sound of the

metronome. The speed of the metronome was the same as

the default speed in the practice phase. Finally, we inter-

viewed the subjects after the examination.

There are three types of keying error: incorrect key-

ing, when the subject presses an incorrect key, as shown in

Figure 7-(a), non-keying, when the subject does not press

a key that the musical staff indicates should be pressed, as

shown in Figure 7-(b), and extra keying, when the subject

presses not only correct keys but also other keys, as shown

in Figure 7-(c).

There are two types of rhythm error: extra rest and in-

correct holding time. Incorrect holding time is when the

subject holds a key over or under the indicated time, tak-

ing into account the time allowed for error. In this evalu-

ation, we define the error margin as plus or minus 0.3sec.

For example, the duration of the sixteenth note is 1.2sec in

the tempo used in the test phase, and the rhythm is deemed

correct if the subject holds the sixteenth note from 0.9sec

to 1.5sec. Moreover, extra rest is when the time from re-

leasing the current key to pressing the next key exceeds

0.6sec.

Some subjects sometimes held keys while searching

for the next keys to be pressed, and released keys ahead of

_330 _331

Table 1. The applicable functions

Proposed method Piano roll method Method without rhythm support

The duration bar Applicable NA NA

Rhythm check function Applicable NA NA

The metronome Applicable Applicable Applicable

Control of the metronome using a foot pedal Applicable Applicable NA

Adjustment of the start point of the metronome Applicable Applicable NA

Presentation of keying position and fingering Applicable Applicable Applicable

Selection of cue points Applicable Applicable Applicable

Displaying a piano roll NA Applicable NA

' NA' stands for not applicable.

(a) Incorrect keying

(e) Correct rhythm (c) Extra keying

(f) Correct rhythm

(b) Non-keying

(g) Correct rhythm because of

next E note is pressed in correct timing

[Correct performance]

[Actual performance]

(c) Extra keying

(f) Correct rhythm

(d) Incorrect holding time

Figure 7. The measurement of rhythm errors and keying

errors

Table 2. The average number of keying and rhythm errors

Keying error Rhythm error

Average SD Average SD

Proposed method 6.0 2.6 10.7 3.8

Piano roll method 31.0 1.4 27.5 3.5

Method without 34.5 0.7 46.0 11.3

rhythm support

* Standard Deviation

the indicated release timing in order to search for the next

keys. We consider this to be not only incorrect pressing

but also extra rest. We judge the rhythm error based on

the duration of an incorrect keying error when the subject

presses an incorrect key as shown in Figure.7-(e). The

case where the subject noticed the keying error and recti-

fied it by pressing the correct key counts as a keying error

but not a rhythm error as shown in Figure 7-(f). Non-

keying is not a rhythm error, as shown in Figure 7-(g).

5.2. Results and Consideration

Table 2 shows the average number of keying and rhythm

errors.

The results show that the proposed method signifi-

cantly enhanced learning effectiveness, when compared

with the piano roll method and the method without rhythm

support. The average number of keying errors and rhythm

errors of the proposed method is small. The difference

between the average number of each error, when compar-

ing the proposed method to the piano roll method and the

method without rhythm support, was at a level of 5%, cal-

culated from Steel-Dwass' multiple comparison test. We

discuss the results relating to the proposed functions as

follows. The behavior of the subjects was observed by the

person overseeing the experiment, who consulted with the

subjects directly after the evaluation.

The duration bar The reason that the subjects who used

the proposed method were able to learn piano performance

effectively is that the comprehension of rhythm and the

reading of a musical staff were improved. The subjects

who used the proposed method or the piano roll method

passed on comments such as that the explicit presentation

of the rhythm helped them to enhance their comprehen-

sion of it.

The subjects using the proposed method were able to

acquire not only the rhythm information but also pitch in-

formation at the same time, as the duration bar is was lay-

ered over the notes of the trial piece. In the beginning

stage of the evaluation, the subjects concentrated on ac-

quiring the keying information presented on the musical

keyboard and rhythm information from the duration bars,

and playing with the correct keying and rhythm based on

the acquired information. Once they were used to playing

the trial piece, they began to understand the connection

between the keying and rhythm information and the nota-

tions on the musical staff, and they became able to read

out the pitch and rhythm information from the musical

notes directly. The subjects using the piano roll method,

in the beginning stage of the evaluation, did not look at

the musical notations on the trial piece, as they practiced

the keying and fingering while looking at the information

presented on the keyboard. Next, they used the piano roll

score to learn the rhythm once they had almost mastered

the keying position and fingering. They could not afford

to look up at the musical staff above the piano roll score.

Finally, they hardly spent any time practicing with only

the musical staff score. As a result, when they performed

the trial piece in the test phase and unknown or difficult

notations appeared, they made a lot of mistakes because

they weren ' t able to read out information from the musi-

cal staff.

The subjects using the method without rhythm support

hardly looked at the musical staff in the beginning stage

of the evaluation. They practiced rhythm with the musi-

cal notation after they had acquired the keying position

and fingering, but they did not have much time to practice

them. As a result, they made many mistakes.

Rhythm check function The rhythm check function con-

tributes to improving the ability to read the musical staff.

The subjects who used the proposed method had to be

conscious of the keying timing and the release timing from

the beginning stage of this evaluation, because if they made

rhythm errors the system did not show the next keys in-

formation. As a result, they paid more attention to the

musical staff.

Regarding the number of rhythm errors, the subjects

rectified incorrect rhythm as they went along, by using

the rhythm check function. Generally, it is difficult for

beginners to time the pressing and releasing of keys for

each hand and to be conscious of their own mistakes. The

subjects made a lot of rhythm mistakes relating to this tim-

ing. The rhythm check function enabled the subjects who

used the proposed method to improve their playing be-

cause they noticed their mistakes and they could practice

the difficult parts again and again.

Control of the metronome using a foot pedal The con-

trol of the speed of the metronome using a foot pedal was

applied to the subjects who used the proposed method and

the piano roll method. However, only two subjects play-

ing with the proposed method used the foot pedal function

because the other subjects focused entirely on the other

functions. The two subjects who used the pedal practiced

the parts which they found easy or difficult in faster or

slower tempo by using this function.

Adjustment of the start point of the metronome In re-

gard to the adjustment function for the start point of the

metronome, the subjects passed on comments such as that

the function was convenient because they did not have to

consider the timing of the metronome before starting to

play. We particularly noticed that the subjects needed a

lot of rests to check keying position and fingering in the

beginning stage of this evaluation. Therefore, subjects

using the method without rhythm support sometimes ig-

nored the click of the metronome because of the added

difficulty of keeping time with it.

Presentation of keying position and fingering on the

keyboard, and selection of cue points All the subjects

used the function that presents keying position and finger-

ing from the beginning stage of this evaluation. Further-

more, the cue point function was also used frequently to

practice difficult areas again and again. We confirmed the

effectiveness of these two functions from the comments

of all the subjects as well.

6. CONCLUSIONS

We constructed a musical staff-based piano learning sup-

port system considering rhythm learning. The learner un-

derstands the duration of notes and rests intuitively by

using the duration bars layered over the notes and rests

on a score. The learner can also understand the remain-

ing time for which they should hold each key by observ-

ing the changing color of the bars. The Rhythm check

function helps users notice their own mistakes and rectify

them. The results of evaluative experiments confirmed

that the subjects using our proposed system played the

trial piece using correct keying and rhythm during the

30 minute training period, and the system significantly

enhanced learning effectiveness in the early stage, when

compared with the piano roll method.

As described in Section 1, playing the piano requires

various techniques, such as correct keying, fingering and

rhythm, which generally need long-term practice. There-

fore, conventional piano learning methods make learners

practice each technique individually, thus beginners often

give up as it takes tremendous time and effort to acquire

the skills needed to play a song adequately. We propose

a comprehensive learning style, which allows learners to

acquire several skills at the same time, by enhancing hu-

man processing ability using multimedia technology and

information design technology. Future work will involve

constructing a more comprehensive piano learning system

that includes not only keying and rhythm information but

also fingering, dynamics, and articulation, and will in-

clude evaluative experiments conducted on beginners of

various generations, as well experiments carried out over

a longer period of time.

7. ACKNOWLEDGMENTS

This research was supported in part by a Grant-in-Aid for

Scientific Research for Young Scientists(B) (21700198)

from the Japanese Ministry of Education, Culture, Sports,

Science and Technology.

8. REFERENCES

[1] CASIO: Lighted Keys Keyboards:

http://www.casio.com/products/Musical Instruments/

Lighted Keys/.

[2] Concert Hands System:

http://www.concerthands.com/.

[3] KAWAI: PianoMaster:

http://www.kawai.co.jp/cmusic/

products/pm/index.htm. (in Japanese)

[4] Keyboardmania

http://en.wikipedia.org/wiki/Keyboardmania.

[5] Roland: http://www.roland.com/piano/DigiScore.html.

[6] A. Butz, M. Schneider, and M. Spassova: Search-

Light A Lightweight Search Function for Pervasive

_332 _333

Environments, Proc. of IEEE International Confer-

ence on Pervasive Computing, pp. 351–356 (2004).

[7] A. Suganuma, Y. Ogata, A. Shimada, D. Arita, and

R. Taniguchi: Billiard Instruction System for Begin-

ners with a Projector-Camera System, Proc. of Inter-

national Conference on Advances in Computer En-

tertainment Technology, pp. 3–8 (2008).

[8] C. Lewiston: MaGKeyS: A Haptic Guidance Key-

board System for Facilitating Sensorimotor Training

and Rehabilitation, PhD Thesis. MIT Media Labora-

tory (2008).

[9] C. Pinhanez: The Everywhere Displays Projector:

A Device to Create Ubiquitous Graphical Interfaces,

Proc. of ACM International Conference on Ubiqui-

tous Computing (2001).

[10] H. Ishii, C. Wisneski, J. Orbanes, B. Chun, and

J. Paradiso: PingPongPlus: Design of an Athletic-

Tangible Interface for Computer-Supported Cooper-

ative Play, Proc.s of ACM conference on Human fac-

tors in computing systems, pp. 394–401 (1999).

[11] K. Huang, E. Y. Do, and T. Starner: PianoTouch: A

wearable Haptic Piano Instruction System for Pas-

sive Learning of Piano Skills, Proc. of IEEE Interna-

tional Symposium on Wearable Computers, pp. 41–

44, (2008).

[12] M. Mukai, N. Emura, M. MIURA, and M. Yanagida:

Generation of Suitable Phrases for Basic Training to

Overcome Weak Points in Playing the Piano, Proc.

of International Congress on Acoustics, MUS-07-

018 (2007).

[13] P. Misty, P. Maes, and L. Chang: WUW - wear Ur

world: a Wearable Gestural Interface, Proc. of ACM

conference on Human factors in computing systems,

pp. 4114–4116 (2009).

[14] R. B. Dannenberg, M. Sanchez, A. Joseph, P. Capell,

R. Joseph, and R. Saul: A Computer-Based Multi-

Media Tutor for Beginning Piano Students, Jour-

nal of New Music Research, 19 (2-3), pp. 155–173,

1990.

[15] R. Raskar, J. V. Baar, P. Beardsley, T. Willwacher,

W. Rao, and C. Forlines: iLamps: Geometrically

Aware and Self-Configuring Projectors, Proc. of In-

ternational Conference and Exhibition on Computer

Graphics and Interactive Techniques (2003).

[16] S. Akinaga, M. MIURA, N. Emura, and Masuzo

Yanagida: An Algorithm to Evaluate the Appropri-

ateness for Playing Scales on the Piano, Proc. of

International Congress on Acoustics, MUS-07-005

(2007).

[17] S. Akinaga, M. MIURA, N. Emura, and Masuzo

Yanagida: Toward Realizing Automatic Evaluation

of Playing Scales on the Piano, Proc. of Interna-

tional Conference on Music Perception and Cogni-

tion, pp. 1843–1847 (2006).

[18] S. Smoliar, J. Waterworth, and P. Kellock: pi-

anoFORTE: A System for Piano Education Beyond

Notation Literacy, Proc. of the Third ACM Inter-

national Conference on Multimedia, pp. 457–465

(1995).

[19] T. Kitamura and M. Miura: Constructing a Support

System for Self-learning Playing the Piano at the

Beginning Stage, Proc. of International Conference

on Music Perception and Cognition, pp. 258–262

(2006).

[20] Y. Takegawa, T. Terada, and T. Tsukamoto: Design

and Implementation of a Piano Practice Support Sys-

tem using a Real-Time Fingering Recognition Tech-

nique, Proc. of International Computer Music Con-

ference (2011).

[21] Y. Takegawa, T. Terada, and S. Nishio: Design and

Implementation of a Real-Time Fingering Detec-

tion System for Piano Performances, Proc. of In-

ternational Computer Music Conference, pp. 67–74

(2006).

MAESTRO: USING TECHNOLOGY TO IMPROVE

KINESTHETIC SKILL LEARNING OF MUSIC CONDUCTORS

School of Music

Georgia Institute of Technology

ABSTRACT

The use of technology in music conductor training is a

growing area of interest. The expressive, subtle, and

meaning- rich gestures that are used in conducting, serve

as fruitful ground for innovative research in areas such

as artificial vision, gesture following , and musical

mapping. While it is known that the kinesthetic skills of

conducting are acquired through hours of intensive

training, practice with real time audio and visual

feedback is severely limited by availability, focus, and

good will of live musicians. The current project, titled

Maestro, builds upon previous work and provides a new

approach for training beginning conductors: a system

allowing the conductor to practice basic to advanced

baton skills accompanied by a virtual orchestra that

responds to the conductor's baton gestures affecting

tempo, duration, articulation, and dynamics. By

incorporating gesture anticipation and tracking, machine

learning for gesture analysis, utilization of physical

modeling for high-quality audio, Maestro provides

immediate feedback that is directly related to subtle

variations of performed conducting gestures.

1. INTRODUCTION

Performing music, whether playing an instrument,

singing, or conducting, requires a combination of aural,

cognitive, and kinesthetic skills that require specific

practice to improve [1], [2]. Such skills could include

learning the fingering patterns of major and minor scales

on a particular instrument or the weight on the bow of a

stringed instrument. Kinesthetic skills are also the

foundation of beginning music conducting skills [3].

Beginning conducting students must learn a plethora of

movements that include instruction on torso, head, and

arm positions and a variety of expressive movements

intended to bring about a response from performers.

The acquisition of such skills is a challenging task,

which is historically achieved with individual or group

instruction, followed by individual practice. Indeed,

several technological innovations address this effort by

putting an emphasis on the development of kinesthetic

skills related to performing music or providing

sophisticated feedback (either in real-time or non real-

time) to act as a virtual music teacher.

Such tools present different solutions for the

practical issues as well as the psychological aspects of

acquiring musical skills. Practicing in front of a teacher,

peers, and eventually an audience may cause different

responses ranging from indifference to anxiety [4], [5].

Creating individualized instructional tools and allowing

more comfortable practicing environments can be

invaluable to many populations that are affected by such

difficulties. We contend that use of the Maestro system

in such traditional learning environments would enhance

the learning experience and encourage kinesthetic

awareness and overall musical skill development.

The project seeks to advance previous conducting

technology and pedagogy through two core advances: a)

the delivery of rich real-time audio and visual feedback

through the Maestro system to enable the refinement of

kinesthetic skills of conducting gestures affecting

variations of speed, articulation, dynamic, and speed,

and b) the ability to practice conducting gestures without

the need for live musicians or peers. The Maestro system

introduces technical innovation-based research in three

main areas: a) gesture anticipation and tracking; b)

machine learning for gesture detection and classification;

c) utilization of physical modeling for high quality,

subtle musical feedback. This work is designed to foster

more opportunities for meaningful learning experiences

through the beginning conductor's discovery of

subtleties of gestures and their effect on musical

performance.

2. RELATED WOKS

In recent years, there have been several attempts to

simulate the conductor's baton. Developments in mobile

technology and the wide availability of sensors and

accelerometers encouraged researchers to explore the

hitherto relatively uncharted realm of conducting. The

Radio Baton [6] was one of the first systems developed

in this field. It offered an interactive conducting

experience by controlling the tempo of a MIDI sequence

as a feedback to the gesture. Other systems in later years

incorporated sensors for more precise input analysis,

such as measuring the pressure on the baton [7], tracking

the conductor's muscle tension [8], and using a built- in

camera on the baton [9]. Improvement over the years

included transition from MIDI to audio-based musical

feedback [10] to more sophisticated and realistic forms

of sound generations [11].

Similar projects targeted simulation of the

conducting experience as a way to experience

controlling an orchestra, rather than for researching the

subtleties of conducting gestures and their musical

effect. In 2004, Borchers offered children the

opportunity to conduct the Vienna Philharmonic

Orchestra. The 'conductor' would stand in front of a

video screen and control the tempo of an actual

performance [12]. Two other systems with similar focus

are iSymphony [13] and Pinocchio [14], developed a few

years later.

Along with programs designed to familiarize and

introduce the conducting experience to non-musicians,

... Takegawa et al. reduce cognitive load by projecting a piano roll directly on the keyboard [9]. However, in order to make this work the authors had to paint the entire keyboard white, and the projector required complex calibration. ...

... Based on previous research [9], [10] we decided to use an AR/MR system and a piano roll in order to remove the need for reading sheet music and for changing view direction between the keyboard and the music sheet or computer monitor. ...

... Our research group has previously constructed a piano learning support system for beginners [2], [3], [4]. It is clear that presenting information directly connected to physical actions such as pressing keys is intuitive for beginners. ...

... Targeting various fields, such as singing, penmanship, instrument-playing, and palpation measurement, learning systems have been proposed which both present exemplary information and current state, and point out mistakes. For example, a lighted keyboard which indicates the next key to be pressed, as well as various kinds of software, are sold by instrument makers for pianists in the early stage of practice [4]. This early stage refers to the stage in which piano beginners first look at a score and practice to remember fingering and keying positions. ...

  • Marimo Kumaki
  • Yoshinari Takegawa
  • Keiji Hirata Keiji Hirata

In this paper, we propose a positioning learning support system with false information and vague information teaching functions for violin beginners. A hypothesis is made that the system obscures presented information from the learner or presents erroneous information so that the learner can break away from the learning support information. In an experiment using this system, it was shown that the group learning with vague information and false information tailored to the proficiency level learned more effectively than the group that continued learning with only accurate learning support information. This demonstrates the usefulness of the system. Moreover, we structured the causal relationship between True mode, Vague mode, and False mode objectively and visually in the proposed system using a structured modeling method using the ISM method (Interpretive Structural Modeling). A mutual relationship between various modes can be illustrated as with a multilayer directed graph. From these results, we consider the ideal learning model for the casual relationship between each mode.

... Task: Instrument performance analysis. One of the main functions of scores is enabling instrumentalists to (re-)interpret musi- Those are implemented as figures, appearing to walk over the keys to be pressed [XTI14], a streamed second player accompanying the user [XAWI13], a rotated piano roll notation moving towards the player [WRR*13, YE13] or visualizations conveying rhythm using colours to indicate how long a note has to be played [TTT12]. In comparison to hinting of what should be played, Smoliar et al. [SWK95] and the game Rocksmith [Ubi11] offer means of validation by indicating the discrepancy of actually pressed keys to the composed notes. ...

Digital methods are increasingly applied to store, structure and analyse vast amounts of musical data. In this context, visualization plays a crucial role, as it assists musicologists and non‐expert users in data analysis and in gaining new knowledge. This survey focuses on this unique link between musicology and visualization. We classify 129 related works according to the visualized data types, and we analyse which visualization techniques were applied for certain research inquiries and to fulfill specific tasks. Next to scientific references, we take commercial music software and public websites into account, that contribute novel concepts of visualizing musicological data. We encounter different aspects of uncertainty as major problems when dealing with musicological data and show how occurring inconsistencies are processed and visually communicated. Drawing from our overview in the field, we identify open challenges for research on the interface of musicology and visualization to be tackled in the future. Digital methods are increasingly applied to store, structure and analyse vast amounts of musical data. In this context, visualization plays a crucial role, as it assists musicologists and non‐expert users in data analysis and in gaining new knowledge. This survey focuses on this unique link between musicology and visualization. We classify 129 related works according to the visualized data types, and we analyse which visualization techniques were applied for certain research inquiries and to fulfill specific tasks. Next to scientific references, we take commercial music software and public websites into account, that contribute novel concepts of visualizing musicological data. We encounter different aspects of uncertainty as major problems when dealing with musicological data and show how occurring inconsistencies are processed and visually communicated. Drawing from our overview in the field, we identify open challenges for research on the interface of musicology and visualization to be tackled in the future.

... Pembelajaran piano pada tingkat awal difokuskan pada latihan untuk meningkatkan keterampilan jari melalui berbagai macam latihan (Takegawa, Terada, & Tsukamoto, 2008). Latihan yang sering digunakan untuk meningkatkan keterampilan jari adalah latihan tangga nada Penggunaan penjarian yang benar merupakan aspek dasar penting pada pembelajaran piano. ...

  • Oriana Tio Parahita Nainggolan Oriana Tio Parahita Nainggolan

Pembelajaran tangga nada merupakan materi dasar dan penting pada pembelajaran piano. Mempelajari tangga nada memberikan manfaat untuk meningkatkan kekuatan dan keterampilan jari dalam bermain piano. Untuk dapat memainkan tangga nada, hal yang harus dilakukan adalah menghafal penggunaan penjarian yang benar. Strategi pembelajaran yang digunakan dalam menghafal penjarian tangga nada adalah dengan mengelompokan penjarian. Pengelompokan penjarian adalah penggunaan penjarian yang sama pada beberapa tangga nada. Penelitian ini dilakukan untuk mengkaji penggunaan strategi pengelompokan penjarian dalam menghafal penjarian tangga nada pada mata kuliah Piano Dasar I. Penelitian ini dilakukan di Program Studi S-1 Pendidikan Musik, Fakultas Seni Pertunjukan, Institut Seni Indonesia Yogyakarta, semester genap tahun akademik 2017/2018. Hasil penelitian menunjukkan bahwa strategi pengelompokan penjarian mempermudah mahasiswa dalam menghafal penjarian tangga nada pada materi Instrumen Dasar I. Scales Fingering Memorization Strategy in the Basic Instrument Course I. Scales play an important role as a fundamental basic to play piano. Studying scales will give students the opportunity not only developing hands coordination but also building the strength of hands and fingers. In doing the scales, the first thing to do is memorizing fingering on every scale, because every scale has their own fingering. The learning strategy to memorize fingering's scales is by grouping the same fingering that is use for several scales. This research aims to study the use of grouping fingering towards students learning achievement in the subject of Instrumen Dasar I at Music Education Study Program, Performing Arts Faculty, Indonesia Institute of the Arts Yogyakarta, second semester of the academic year 2017/2018. As the result of this study, it can be concluded that grouping fingering help the students to memorize scale's fingering in learning scales.Keywords: piano; fingering; basic instrument

  • Hokuto Tsutsumi
  • Hiroaki Nishino

Learning and improving piano play performance supported by an ICT-based training system is a difficult and time consuming task. It is especially difficult for beginners because existing technologies force users the continuation of excessive burdens on their vision and concentration. We propose a piano performance training system where the users can easily focus on the acquisition of piano performance skills with lighter cognitive loads. The proposed system provides a training environment based on visual and tactile guidance. It enables the users to keep their eyes on a screen for checking the performance instructions presented on it and concentrate on typing keys with correct fingers according to the instructions. It also uses the tactile stimuli for intuitively prompting the users to hit the keys with correct fingers on a real keyboard. The system recognizes the users' fingering, judging whether they hit the right keys with correct fingers, and feedback the results via tactile sensation. It enables even beginners to easily practice piano performance with correct fingering by self-study.

  • Kazushige Ashimori
  • Hiroshi Igarashi

Haptics is one of effective interface giving information as a position, force, and object texture. When people learn to motion like playing musical instruments, visual, hearing, and haptics feedback are valid. In this paper, we proposed haptic assist on musical instruments playing. As a method of haptic presentation, a haptic glove with bilateral control is developed. The glove has some small geared motors for each finger joint. As a result, the proposed method showed the usefulness of an intuitive learning teaching system by haptic feedback.

Purpose: Patients with cerebral palsy with severe motor disabilities are limited not only in everyday life activities but also in choice of their hobbies. Playing a musical instrument is for the majority not possible, even though music constitutes a central component of many relaxation activities for people with disabilities. To give affected patients the opportunity to make music and to learn piano playing through somatosensory perception, a prototype of piano jacket was developed. Method: A cycling jacket was equipped with boxes, each representing a musical note, incorporated into the sleeves. Each box contains vibration motors and LEDs. These can be used to translate the sequences of piano key presses performed at an external E-piano. An additional operation mode allows the user to actively play a melody himself by touching the same boxes that also incorporate touch sensor components. Results: A working prototype of the piano jacket was developed. Conclusion: The sensory piano jacket provides patients with cerebral palsy and contractures the possibility to develop sensorimotor skills, motor abilities and participation in music-related activities. The jacket is planned to be used in music lessons. • Implications for Rehabilitation • The system is suitable even for patients with severe motor disabilities and especially joint contractures. • Excellent handling through large easily accessible communication buttons. • All building blocks are held in a garment, which allows for easy portability and gripping comfort.

  • Roger B Dannenberg Roger B Dannenberg
  • Marta Sanchez
  • Annabelle Joseph
  • Ronald Saul

The Piano Tutor provides computer‐based instruction to beginning piano students. Intended as a supplement to traditional instruction, the Piano Tutor helps students by correcting mistakes before they become ingrained through practice and by teaching new material as soon as the student is ready. The Piano Tutor combines an expert system with state‐of‐the‐art music recognition software and multimedia output devices to provide a stimulating learning environment that tailors instruction to the student's needs.

Projectors are currently undergoing a transformation as they evolve from static output devices to portable, environment-aware, communicating systems. An enhanced projector can determine and respond to the geometry of the display surface, and can be used in an ad-hoc cluster to create a self-configuring display. Information display is such a prevailing part of everyday life that new and more flexible ways to present data are likely to have significant impact. This paper examines geometrical issues for enhanced projectors, relating to customized projection for different shapes of display surface, object augmentation, and co-operation between multiple units.We introduce a new technique for adaptive projection on nonplanar surfaces using conformal texture mapping. We describe object augmentation with a hand-held projector, including interaction techniques. We describe the concept of a display created by an ad-hoc cluster of heterogeneous enhanced projectors, with a new global alignment scheme, and new parametric image transfer methods for quadric surfaces, to make a seamless projection. The work is illustrated by several prototypes and applications.

  • Yoshinari Takegawa
  • Tsutomu Terada Tsutomu Terada
  • Shojiro Nishio

Fingering is an important aspect of piano performance be-cause it affects a pianist's musicality. If audience members, especially pianists, shared the real-time fingering of the per-former in a concert, they would feel a sense of togetherness, and this could help them in learning techniques of profes-sional piano performance. To realize such a possibility, the goal of our study is to construct a real-time fingering de-tection system for piano performance. Our system achieves real-time fingering detection by integrating a simple camera-based image-detection method and musical rules. We have developed a prototype system and evaluated in effectiveness by actual use.

  • Stephen W. Smoliar
  • John A Waterworth John A Waterworth
  • Peter Kellock

This paper addresses the problem of how to automatically generate visual representations of recorded histories of distributed multimedia collaborations. The work reported here focuses mainly on what we consider to be an innovative approach to this problem, ...

We present PianoTouch, a wearable, wireless haptic piano instruction system, composed of (1) five small vibration motors, one for each finger, fitted inside a glove, (2) a Bluetooth module mounted on the glove, and (3) piano music output from a laptop. Users hear the piano music and feel the vibrations indicating which finger is used to play the note. We investigate the system's potential for passive learning, i.e. learning piano playing automatically while engaged in everyday activities. In a preliminary study, four subjects learned two songs initially and then wore the PianoTouch glove for 30 minutes while listening to the songs repeated. One of the songs included tactile sensations and the other did not. The study found that after 30 minutes, the PianoTouch subjects were able to play the song accompanied by tactile sensations better than the non-tactile song. These results suggest the value of a more detailed study.

  • Seiko Akinaga
  • Norio Emura Norio Emura
  • Masanobu Miura
  • Masuzo Yanagida

The aim of this study is to realize automatic evaluation of playing scales on the piano and to point out defects of the user in his/her playing scales. Records of playing scales together with corresponding subjective scores are collected in order to determine the evaluation criteria. Performances were recorded on a MIDI sequencer. Onset intervals, ve-locities, and durations for each note are recorded and a set of three regression curves for each performance is calcu-lated by spline interpolation for the average value of each parameter calculated for each wrist position on the key-board. The scores obtained are employed as both test data or training data in turn to achieve open tests. KL expansion and k-NN algorithm are tried to predict subjective scores of new scale-playing data. Outputs by the proposed system are relatively similar to subjective evaluation scores and the appropriateness of values for description parameters is discussed.

  • Tamaki Kitamura
  • Masanobu Miura

This paper aims at realizing a support system for self-learning playing the piano at the beginning stage. The sys-tem is conceptually designed as a computer-based interac-tive system, and it is expected that using the system non-pianists of wide age ranges are able to learn playing the piano without experts' instructions. Designed here is an original procedure, strongly dependent to a famous text-book "Methode Rose", and the ways of producing exercises described in "Bayer" are interpreted manually to be in-cluded. The proposed system will be implemented by em-ploying this designed procedure. Developed here is a sub-system, which automatically generates basic exercises of learning observing records of his/her weak points. Other facilities of it are: a) analyzing his/her performance re-corded using a MIDI sequencer, b) extracting inaccuracies and/or deviations concerning timing, MIDI-velocity and duration, and c) showing obtained defects of users to a PC-display together with optimum exercises corresponding to their errors.

  • Pranav Mistry
  • Pattie Maes Pattie Maes
  • Liyan D. Chang

Information is traditionally confined to paper or digitally to a screen. In this paper, we introduce WUW, a wearable gestural interface, which attempts to bring information out into the tangible world. By using a tiny projector and a camera mounted on a hat or coupled in a pendant like wearable device, WUW sees what the user sees and visually augments surfaces or physical objects the user is interacting with. WUW projects information onto surfaces, walls, and physical objects around us, and lets the user interact with the projected information through natural hand gestures, arm movements or interaction with the object itself.